Showing posts with label CICD. Show all posts
Showing posts with label CICD. Show all posts

Try Data Engineering in Python with Snowpark in Just 10 Minutes [new feature]

Try Data Engineering in Python with Snowpark


Getting Started with Snowpark in Snowflake Python Worksheets

By completing this guide, you will be able to try Snowpark for Python all from within the Snowflake UI. At the end, you’ll have a better understanding of how to perform core data engineering tasks using Snowpark in Snowflake Python Worksheet.
What is Snowpark
It allows developers to query data and write data applications inside Snowflake with languages other than SQL using a set of APIs and DataFrame-style programming constructs in Python, Java, and Scala. These applications run on and take advantage of the same distributed computation on Snowflake's elastic engine as your SQL workloads. Learn more about Snowpark.
What is Python Worksheet
Python worksheets are a new type of worksheet in Snowsight that helps you get started with Snowpark faster. Users can develop data pipelines, ML models and applications directly inside Snowflake, no additional IDE (development UI) to spin up, set up or maintain for Python. These worksheets can be converted into procedures to schedule your Snowpark applications.
What you will learn
- Load data from Snowflake tables into Snowpark DataFrames

- Perform Exploratory Data Analysis on Snowpark DataFrames

- Pivot and Join data from multiple tables using Snowpark DataFrames

- Save transformed data into Snowflake table
What you will build
A prepared dataset that can be used in downstream analysis and applications. For example, training a machine learning model.


Setup Lab

You may start with a 30-day trial account ($400 credit).






















End of Line Matters for Shell Script in CICD Pipeline

End of Line Matters for Shell Script in CICD Pipeline

You may get an error or failed pipeline if the end of line of a shell script file is CRLF (\r\n) not LF (\n) 

LF

CRLF


In VS Code, you may convert the End of Line from CRLF to LF. Or you may change the default setting



Introduction to Shell Script on UNIX (1)

Introduction to Shell Script on UNIX (1)

Shell解析器


可以通过以下命令查看系统所提供的解析器

$cat /etc/shells

HelloWorld example (helloworld.sh)

创建脚本程序

$touch helloworld.sh
$vi helloworld.sh
Edit:
#!/bin/bash
echo "helloworld"

or $cat > helloworld.sh
#!/bin/bash
echo "HelloWorld!" 


Note:

First line: #!/bin/bash

Exit cat > by ctrl + c

脚本的执行方式

1 bash/sh + 脚本的相对路径/绝对路径

sh

$sh ./helloworld.sh


$sh /home/qfhou/bash_study/helloworld.sh 

Similarly for bash

$bash ./helloworld.sh

$bash /home/qfhou/bash_study/helloworld.sh

2 采用输入脚本的相对路径/绝对路径执行脚本 (脚本需要具有可执行权限)

赋予helloworld.sh可执行权限
$chmod +x ./bash_study/helloworld.sh

Note: 

Search cmd history

ctrl + r + string

(reverse-i-search)`ch': chmod +x ./bash_study/helloworld.sh

Full history

$history

执行脚本

$./helloworld.sh

$/home/qfhou/bash_study/helloworld.sh

变量

定义

$变量名=变量值  (=前后不能有空格)

显示变量值

$echo $变量名

撤销变量
$unset 变量名 (not unset $变量名)

静态变量声明(不能撤销)
$readonly B=9

把变量提升为全局环境变量,可供其他 Shell 程序使用

$export 变量名

特殊变量

$n
功能描述:n 为数字,$0 代表该脚本名称,$1-$9 代表第一到第九个参数,十以
上的参数,十以上的参数需要用大括号包含,如${10}

$#
功能描述:获取所有输入参数个数,常用于循环 , 判断参数的个数是否正确以及
加强脚本的健壮性

parameter.sh


Results


$*

(功能描述:这个变量代表命令行中所有的参数,$*把所有的参数看成一个整体)

$@

(功能描述:这个变量也代表命令行中所有的参数,不过$@把每个参数区分对待)

Test (parameter.sh)



Output




$?

(功能描述:最后一次执行的命令的返回状态。如果这个变量的值为 0,证明上一
个命令正确执行;如果这个变量的值为非 0(具体是哪个数,由命令自己来决定),则证明
上一个命令执行不正确了。)

Note: True -> 0; False -> 1

Last execution:





What are horizontal and vertical scaling in data engineering

Horizontal and vertical scaling

This pic tells you the difference between vertical and horizontal scaling.


How to get into a Docker container

How to get into a Docker container


When you use docker image for your development, you may want to get into the container to check the files etc. Here are three ways to get into the container for your image.

|=== 1. Terminal
|=== 2. Docker + Terminal
|=== 3. Docker

Solution 1 (Terminal):

- check the containerID with $docker ps
- then  run $docker exec -it containerID /bash/bin

Solution 2 (Docker + Terminal):

- if you are using docker desktop, open it and check the running docker container
- you would be able to see >/ (CLI) when you move your mouse over the container
- click and go for it


Solution 3 (Docker):

-> Click the name of your container in Docker

-> Then, click Files



-> You can browse, open and edit files as you want

Enjoy!

Top 20 Git Commands

git config

Usage: git config –global user.name “[name]”

Usage: git config –global user.email “[email address]”

This command sets the author name and email address respectively to be used with your commits.

git init

Usage: git init [repository name]

This command is used to start a new repository.

git clone

Usage: git clone [url]

This command is used to obtain a repository from an existing URL.

git add

Usage: git add [file]

This command adds a file to the staging area.

Usage: git add *

This command adds one or more to the staging area.

Usage: git add .

This command adds one or more to the staging area.

git commit

Usage: git commit -m “[ Type in the commit message]”

This command records or snapshots the file permanently in the version history.

Usage: git commit -a

This command commits any files you’ve added with the git add command and also commits any files you’ve changed since then.

git diff

Usage: git diff

This command shows the file differences which are not yet staged.

Usage: git diff –staged

This command shows the differences between the files in the staging area and the latest version present.

Usage: git diff [first branch] [second branch]

This command shows the differences between the two branches mentioned.

git reset

Usage: git reset [file]

This command unstages the file, but it preserves the file contents.

Usage: git reset [commit]

This command undoes all the commits after the specified commit and preserves the changes locally.

Usage: git reset –hard [commit]

This command discards all history and goes back to the specified commit.

git status

Usage: git status

This command lists all the files that have to be committed.

git rm

Usage: git rm [file]

This command deletes the file from your working directory and stages the deletion.

Git log

Usage: git log

This command is used to list the version history for the current branch.

Usage: git log –follow[file]

This command lists version history for a file, including the renaming of files also.

git show

Usage: git show [commit]

This command shows the metadata and content changes of the specified commit.

git tag

Usage: git tag [commitID]

This command is used to give tags to the specified commit.

git branch

Usage: git branch

This command lists all the local branches in the current repository.

Usage: git branch [branch name]

This command creates a new branch.

Usage: git branch -d [branch name]

This command deletes the feature branch.

git checkout

Usage: git checkout [branch name]

This command is used to switch from one branch to another.

Usage: git checkout -b [branch name]

This command creates a new branch and also switches to it.

git merge

Usage: git merge [branch name]

This command merges the specified branch’s history into the current branch.

git remote

Usage: git remote add [variable name] [Remote Server Link]

This command is used to connect your local repository to the remote server.

git push

Usage: git push [variable name] master

This command sends the committed changes of master branch to your remote repository.

Usage: git push [variable name] [branch]

This command sends the branch commits to your remote repository.

Usage: git push –all [variable name]

This command pushes all branches to your remote repository.

Usage: git push [variable name] :[branch name]

This command deletes a branch on your remote repository.

git pull

Usage: git pull [Repository Link]

This command fetches and merges changes on the remote server to your working directory.

git stash

Usage: git stash save

This command temporarily stores all the modified tracked files.

Usage: git stash pop

This command restores the most recently stashed files.

Usage: git stash list

This command lists all stashed changesets.

Usage: git stash drop

This command discards the most recently stashed changeset.

Featured Posts

SnowPro Badges and Certificates

SnowPro Badges and Certificates Online Verification https://achieve.snowflake.com/profile/richardhou888/wallet

Popular Posts Recommended