Lambda function scheduled to perform backups of another server on s3

Transcript

okay so we have defined the resource that we want this

s3 bucket and now we're ready to deploy it so what

else is there to take into account so um normally the

deploy command would be something like this i'm gonna add it to the make file

so we run it from the folder of the rest of the command comments we could just

see the manually into it um and then we're gonna run like cdk

and then yeah let me explain this basically it

would be something like this deploy it right so um

it's cdk deploy right so why this profile

so you don't need this but basically it could be like this cdk deployed

uh and but and this will create all the resources we create

from our code the cloud formation resources and then

uh interpret that and create for real the resources

also careful with this because this could have real costs right

um so far it's going to be an empty bucket so the costs

will be zero or very minimal but what's with the profile so this

profile is the the user that the cdk is going to use to

get the credentials so basically you will have a credentials

file um like on on on your home directory like on

aw yeah so in this [Music]

yeah so basically in this file credentials which i cannot show you for

obvious reasons but basically let me open it in a new

let me just check it in a in a hidden window so i can

yeah so basically this has a structure like this

that's the second how can i show you this let me just open

another folder for a second yeah so basically this has a structure

so this looks like it's like a config file right so it's

like my user and then this like aws access

piv equals and a very long string and the same thing with like uh

secret secret access key and another long

string and then this is for my user but in our

case we have one that looks like this from backup

s3 and this is why the profile you pass will be

this one right so if you have my user you can choose here to pass my user

maybe you have just a default one there's also

one at the end called default all right so if you don't pass anything

it will be the default one and this allows you to have different

credentials for different things which is a good idea

not to have like a super credential doing doing everything

um yeah so that's all with this so in my case i have a profile called

chrome backup s3 just for this project right

so that's why i'm passing it i'm gonna leave this

the if you hear the comment [Music]

okay um and this should be enough but there's

one more thing to keep into account and we're ready to deploy um the thing

is there is basically if we go to lambda

here you can see here some of my

um sorry not lambda but s3 you can see here some of my buckets these are for

other projects but basically this one that you see here

cdk tool stick toolkit and this is basically the

boilerplate that amazon web services needs so they

need like a packet with some basic assets

and this is what the amazon uses right so um

because of this we're gonna need this basically for this project for lambda

for s3 so this is ways that we just added so

you can just add basically we're gonna add

here the same but bootstrap with the chrome backup s3 because mine

already exists i can already add the name

sometimes it's a bit buggy so let's say this

in this you can run several times and it's not a problem

you will just say it already exists let me keep going

but there's a chance that if you delete it if you read it sometimes

it loses a bit track of if it exists or not

and then you might get an error saying this specific

name doesn't exist in that case you can add

these right otherwise it should be enough

with this part so you just want to run basically a cdk bootstrap

so the cdk creates the minimum thing it needs to run the cdk

and then cdk deploy in my case i have i have it already

so because of that i'm going to put here the same name that we have

yeah it ends in osg osg but it should be the same for you so

basically let's deploy this and we should see an s3 bucket created

right an s3 packet with this name chrome backup s3

let's run it i'm just gonna run make deploy and we'll run all of this

okay [Music]

and let's see it's running the bootstrap first

uh okay and we have an error with the account access

well so the problem was that we didn't have this file

right um because i'm here in a restricted environment

um like using docker for this tutorial but basically i just created it and put

my credentials here and now it should work so we just do

make deploy okay so the credentials went through

it's about traveling the environment as you see here if the environment already

exists it will say this no changes

and now it's creating the cloud formation change set so from our code

we'll generate cloud formation and that will be created right so this

is looking good and we're gonna see how this will create

in the ui also the the packet so our idea is that

using the same process we're going to create s3 packets but also

lambdas basically anything we need from amazon web services infrastructure will

create this way we'll see there's some pitfalls to take

into account but basically it worked right it says

everything went fine this check is the confirmation

and that's it so we can go back to the ui

and then we should see here our packet i'm reloading

and here's our bucket right it's just empty

but uh but this is our bucket so this is looking good

um now um there is okay let's try so we saw there

is like a a deploy a command right there's also

a destroy command so the destroy command just removes

all the resources that we that we allocated

so let's try this because this is a common problem

and we want to deal with it as soon as possible so basically

is the same but it just says destroy right

i just created this for convenience to just do make destroy but basically is

this the same with the profile if we want to

use one and then there's also an option that is like force basically

it asks for confirmation so force always says yes basically

so let's run make destroy now or cdk destroy and see what happens

so we have the s3 bucket here and what we want what this should do is

remove the bucket right so let's try and make this through

and now it's running because of the force option it's not even gonna ask for

confirmation okay it says destroy it right this stack

was destroyed let's see if that's true so we go here to the ui

and surprise this bucket is still there so um this is a common problem with ss3

packets that they're not deleted basically so we need

to come here and delete the bucket

um also it's quite a manual process because

this interface asks us for things like this which is a good thing you know

but if every time we want to delete resources we need to do this uh

i mean now we i had one bucket but imagine we have a hundred

no so we need to come here and manually do all of this so

it's uh it's not uh it's not how we want to do it

basically so um and the message here says that it has been destroyed right

which is it's not true so um we're gonna find a

solution for for this so what is the solution uh for

this bucket there's another bucket that we can use

from a third party that basically has the correct behavior

so um [Applause]

it's called uh [Music]

how to delete bucket so we can basically let me find that

let me show you the repository that basically is this

a little bugger exactly so this one yeah and it will be empty before it's

destroyed because that's another issue if you have

things inside the bucket you cannot remove it without

removing the things inside so basically quite complicated

but uh once once you add this basically we can copy this import

and basically this is a drop-in replacement right so super easy

so we decided here we need to install it soon also

but yeah basically this it will be auto delete bucket

and all the options inside are the same so

we can just remove this and here we'll use auto delete bucket

and just like that this will work so this is a pretty good library

so i'm very happy with it and um [Music]

yeah i just found it and it's it's really really useful

so um we need to install this first right so we're gonna add it

to our install commands let's go to make file

now we start seeing why why they make file i think it's a good idea

because we can have all this history right

of what we installed and what not and there's quite a lot of stuff so

then we can run this local install devs again

and we should be good to go also while we're at it here there's a make file

and there's some basically file has like an old history it's it's

assuming that we are creating files as a result of this

because this was used for compiling mostly

so basically there's this tiny thing called like phony targets you can find

more about it online but

but basically it's something we can use to make sure that there's no

extra mistakes so here you have some some stuff um this is what you would

have to google if you want to check more about it

um but i mean i mean you have all the resources here too

i just won't bore you with it but basically

you can you can basically here list the targets

where um you let's say normally a makefile will

generate a file and if the file has the same name as the target

then the target is considered executed so you couldn't execute again

right here we don't want anything like that so one thing we can do is just list

all the targets here in this special declaration it's phony

and the only thing this will do is that if for instance if we are running deploy

and there's a file named deploy normally it would not run it because you

could say the file already exists because of this historical accident

right um but we want it to run no matter what

so we don't care if there's a file with the same name

so we just add here all the all the targets now and then we make

sure that it's running every single time what we don't want is that by chance

there's a file called local init or destroy

and then we think we're running it and it's not really running so just as a

site there's this funny thing for make fast i'll just add it here

um and then we can uh it's not needed now but you know uh it

was time to add it at some point so i said it now

and then we can run the locally installed devs so

i'm just gonna run make local install depth

and they should install our new auto delete bucket and we're gonna try it

right now i think it's worth to know about this

library because it's really really useful and it can save you money if you

have a lot of resources and you think you deleted them but you did not delete

them like you will get charged for that that

storage even if it's only a bit but yeah you will so

okay major version we can update this later for now let's keep going so now we

have this installed we're gonna see exactly

this warning disappeared so that's good and now we should be able to do the same

thing as before but now the destroy method should work

right so let's uh let's see first we're gonna

deploy as always this bootstrap thing should do

nothing okay so there's some there's some kind

of mistake so here we must have done some mistake

how to delete bucket from mobile pose

okay so it looks like maybe it could be this ad here it should be

installed i think um here it is npm app

let's try it with this install and see if that works because it seems

like it's the dependencies right so we run local

install devs depths again

and we can see if it works okay it seems now installing new stuff

so let's see [Applause]

and then we try to deploy the bootstrap should be fine

no still we have an error require stack i think it provides some

dependencies of how to delete bucket itself right

so just okay so the error was here all the time right

so i need to read it um it says cannot find module

uh lambda so basically we are not using lambda yet in our

script but this uh [Music]

this auto delete bucket is using it right so we need to add it

for it because it will use lambda to to actually check that we are delete to

delete the packet afterwards right so we can just add it uh

[Music] let me just so basically this one

all right and then we can just add it to the

imports and install it and everything so um reside here and it's gonna be

our quote so we're single [Music]

so this is the lambda one and then we're gonna need to

add it also here to the make file right so here

it's aws and this should be enough now uh

this here there's a warning line and a squiggly line

um yeah so let's just run the install again

because we we added this right and now we have

these new dependencies so let's run the install

now this will install lambda and and the auto delete bucket depends on

lambda so the deploy now should work okay it's doing its thing the lambda was

installed we can see here and the squiggly line should be gone

yeah and then we can deploy let's see if this one worked

it should and they should create the s3 back

okay okay so we have all other dependencies right so now

let's add the cloud formation which also makes sense i'm not sure why

this library is not including all visas dependencies and they're just getting

installed and unless maybe i'm making some kind of

mistake uh yeah but basically aws

cloud formation i wonder if i i don't think i even need

to import this i just need to install it right

okay let's try all these dependencies um transformation

you cannot find module aws cdk cloud formation

okay let's add it uh [Music]

it's a bit strange maybe it's part of another model too anyway

and we're gonna add it so uh i'm not sure we need to import this but

it needs to be installed for sure so let's try to install it only

and then if it complains so this will be the same as before npm

install and cloud formation this time we're just

gonna run this one command and we try it again so we go into the

chrome backup run this command

okay and let's try the deploy again and see what new error

from the parent and see what new error we get

okay it seems like it's going through uh okay do you wish to deploy these changes

yes so now it's asking us right um we can also make this default right

to always say yes i think for the destroy i'm going to

leave it on the default to say yes but in this one no

but just so you know you could add uh maybe let's add it even to the

argument well let's say yes first and now

while it's going on i'll show you all the things

that we can do here um just a second so basically

deploy yeah so there's this option [Music]

uh it's called required approval never

um yeah i mean there's a lot of websites talking about the same thing

but basically approval exactly so this is the the

option so if you want we can add here to the

deploy like require i have it here somewhere

i require approval never so if you don't want this

confirmation which i recommend to leave it on

because this has real world consequences but if you don't want it you can just

add require approval never

so let me just add here the comment and then i'm gonna comment it out but

it's here for documentation okay so this could be the require

approval never right uh so so far it's running

let's see how it's going it's almost finished

and here it is it says it all went correctly

okay so let's see if it's true we can go here the management console in

s3 and we can see our bucket here also

empty because we only created the bucket itself

but it created so that's good um now until here is the same we had before

right so what's the difference the difference is that

when we run destroy this should remove the

the bucket now which would be something different from the default behavior

so let's do that let's run make destroy [Music]

destroy which will not ask for confirmation and now i

should say ideally that everything is okay and then

we should go and check that actually the packet has been deleted so let's see

it's running here's our bucket all right there's some auto bucket stuff here

going on and it says destroy it um so let's see in the ui

i reload the page and our packet is destroyed so that's uh

that's pretty cool so like this we can make sure that our

resources are created and destroyed correctly and we

can build on top of this so yeah this is how we can do it also

and i didn't basically i have to write permissions

here but let me copy here some permissions that you

might need so basically i'm going to add them here

in the documentation but basically to deploy you need certain

permissions i'm just going to copy them here

so some of these might apply to you right so if you use lambda

you will need some kind of land access if you

want to use the logs of cloud watch which we will use

if you want to use s3 you're gonna need s3 access

so you will need to configure this as an access so somewhere in amazon

web services you're gonna need to configure

um the permissions right so somewhere here like users maybe user

groups security groups wait a second

yeah exactly this iam right so here you will be able to so here in user

roles you can add like the the needed permissions right so

probably as roles better but um you can search there and you can add the

permissions that it asks you to add i would recommend having each of these

things in like a different role a different user and

things right it's a bit annoying to add all this uh

all these permissions but you don't want to be using your root user and then

losing it or you know it would be catastrophic

because it can cost a lot of money on amazon so

just that as an aside but um where we are now is we can deploy and we

can destroy resources defined with the cdk

so we're gonna take it from here and start adding here everything we need to

do a chrome right so we're gonna need some lambdas we

need to connect to some servers so the next steps are building up

on this