![]() $ pulumi config set clusterIdentifier my-redshift-cluster name, cluster_type = "single-node", publicly_accessible = False, skip_final_snapshot = True, vpc_security_group_ids =, iam_roles =, ))Īt the top of the program, you’ll see that the code assumes you’ve set several Pulumi configuration values - clusterIdentifier, clusterNodeType, and others - so take a moment to do that now with the Pulumi CLI as well: ClusterArgs ( cluster_identifier = cluster_identifier, database_name = cluster_db_name, master_username = cluster_db_username, master_password = cluster_db_password, node_type = cluster_node_type, cluster_subnet_group_name = subnet_group. s3", route_table_ids =, )) # Create a single-node Redshift cluster in the VPC. require ( "region" ) // Create an S3 bucket to store some raw data.Ĭonst eventsBucket = new aws. Config ( "aws" ) const awsRegion = providerConfig. requireSecret ( "clusterDBPassword" ) // Import the provider's configuration settings.Ĭonst providerConfig = new pulumi. require ( "clusterDBUsername" ) const clusterDBPassword = config. require ( "clusterDBName" ) const clusterDBUsername = config. require ( "clusterNodeType" ) const clusterDBName = config. require ( "clusterIdentifier" ) const clusterNodeType = config. Config () const clusterIdentifier = config. Import * as pulumi from import * as aws from // Import the stack's configuration settings.Ĭonst config = new pulumi. As always, make sure you’ve installed Pulumi and configured your AWS credentials in the usual way first, then run the following commands to get going: So to pick up from there, let’s start by bootstrapping a new project with the code from the previous post. When we left off, we’d gotten Redshift up and running, and we were able to pull the data from S3 into Redshift directly (by running a manual query in the Redshift console), but that’s as far as we got - no automation, no protection from duplicate records, just the absolute basics. In that post, the situation was such that some hypothetical application was generating “events” - little bits of JSON, essentially - and writing them periodically to a text file in S3, and as data scientists, we needed some way to gather up all of these events and load them into a Redshift cluster in order to analyze them later. If you haven’t already, I’d encourage you to read through the previous post to get up to speed on what we’re building and why. So without further ado, let’s finish things off by using AWS Glue (and Pulumi of course) to set up a full ETL pipeline that loads all of our S3 data into Redshift automatically. There’s a lot more you can do with AWS Glue than what we need for this project, but as you’ll see, it’s also an excellent fit. With Glue, you can define processes that monitor external data sources like S3, keep track of the data that’s already been processed (to avoid corruption from duplicate records, as we saw in the previous post), and write code in general-purpose programming languages like Python to process and transform the data on its way into Redshift. When your platform of choice is Amazon Redshift, those questions will often be answered by pointing you to another Amazon service (actually a collection of services) called AWS Glue. These are the kinds of questions you’ll almost always have when setting up a data-processing (or ETL) pipeline - and every platform tends to answer them a little differently. What are our options for loading data automatically - for example, on a regular schedule?.How can we transform the data during the ingestion process?.How do we avoid importing and processing the same data twice?.That went well, but you may recall that at the end of that post, we were left with a few unanswered questions: In our last episode, Deploying a Data Warehouse with Pulumi and Amazon Redshift, we covered using Pulumi to load unstructured data from Amazon S3 into an Amazon Redshift cluster. Announcing OIDC Support for Pulumi Azure Providers.Converting Full Terraform Programs to Pulumi.Enhanced search & Navigation: The new Pulumi Docs experience.Announcing the Terraform Migration Offer.Property Search: Enhanced Resource Search in Pulumi Cloud.Review Stacks: Collaborate in the Cloud.Dependent Stack Updates with Pulumi Deployments. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |