Bash

I made grebuntu to merge all of the separate scripts into one script for all distros, but it doesn't work. The individual scripts do, I tested in VMs. What could have caused the issue? the script in question is tsubuntu.sh btw, can be found in the repo The original scripts are available at https://github.com/Tsu-gu/tsubuntu

Hi all,

I'm trying to put a command together to download a bz2 archive containing an img file and decompress it immediately, basically without saving it to the filesystem. Can this be done?

This is what I've come up with so far, but it's incomplete: wget -qO- "https://opnsense.com/.../img.bz2" | bzip2 -dv

Background: Trying to install OPNsense on Linode. Their hacky official guide says the best way to install FreeBSD is via the rescue mode. But FreeBSD posts their images as .img, so the filesystem size limitation of 1GB for the rescue image isn't an issue. But with OPNsense I need to decompress it.

I have a few different options on how to install this but I see it as a good reason to learn more about stdin/out, piping commands, etc.

Thanks in advance.

Edit

After a long process of roaming the web, re-runs and troubleshoot the script with this wonderful community, the script is functional and does what it's intended to do. The script itself is probably even further improvable in terms of efficiency/logic, but I lack the necessary skills/knowledge to do so, feel free to copy, edit or even propose a more efficient way of doing the same thing.

I'm greatly thankful to @AernaLingus@hexbear.net, @GenderNeutralBro@lemmy.sdf.org, @hydroptic@sopuli.xyz and Phil Harvey (exiftool) for their help, time and all the great idea's (and spoon-feeding me with simple and comprehensive examples ! )

How to use

Prerequisites:

• parallel package installed on your distribution

Copy/past the below script in a file and make it executable. Change the start_range/end_range to your needs and install the parallel package depending on your OS and run the following command:

time find /path/to/your/image/directory/ -type f | parallel ./script-name.sh

This will order only the pictures from your specified time range into the following structure YEAR/MONTH in your current directory from 5 different time tag/timestamps (DateTimeOriginal, CreateDate, FileModifyDate, ModifyDate, DateAcquired).

You may want to swap ModifyDate and FileModifyDate in the script, because ModifyDate is more accurate in a sense that FileModifyDate is easily changeable (as soon as you make some modification to the pictures, this will change to your current date). I needed that order for my specific use case.

From: '-directory<$DateAcquired/' '-directory<$ModifyDate/' '-directory<$FileModifyDate/' '-directory<$CreateDate/' '-directory<$DateTimeOriginal/'

To: '-directory<$DateAcquired/' '-directory<$FileModifyDate/' '-directory<$ModifyDate/' '-directory<$CreateDate/' '-directory<$DateTimeOriginal/'

As per exfitool's documentation:

ExifTool evaluates the command-line arguments left to right, and latter assignments to the same tag override earlier ones.

#!/bin/bash

if [ $# -eq 0 ]; then
    echo "Usage: $0 <filename>"
    exit 1
fi

# Concatenate all arguments into one string for the filename, so calling "./script.sh /path/with spaces.jpg" should work without quoting
filename="$*"

start_range=20170101
end_range=20201230

FIRST_DATE=$(exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$filename" | tr -d '-' | awk '{print $1}')

if [[ "$FIRST_DATE" != '' ]] && [[ "$FIRST_DATE" -gt $start_range ]] && [[ "$FIRST_DATE" -lt $end_range ]]; then
        exiftool -api QuickTimeUTC -d %Y/%B '-directory<$DateAcquired/' '-directory<$ModifyDate/' '-directory<$FileModifyDate/' '-directory<$CreateDate/' '-directory<$DateTimeOriginal/' '-FileName=%f%-c.%e' "$filename"

else
        echo "Not in the specified time range"

fi

Hi everyone !

Please no bash-shaming, I did my outmost best to somehow put everything together and make it somehow work without any prior bash programming knowledge. It took me a lot of effort and time.

While I'm pretty happy with the result, I find the execution time very slow: 16min for 2288 files.

On a big folder with approximately 50,062 files, this would take over 6 hours !!!

If someone could have a look and give me some easy to understand hints, I would greatly appreciate it.

What Am I trying to achieve ?

Create a bash script that use exiftool to stripe the date from images in a readable format (20240101) and compare it with an end_range to order only images from that specific date range (ex: 2020-01-01 -> 2020-12-30).

Also, some images lost some EXIF data, so I have to loop through specific time fields:

• DateTimeOriginal
• CreateDate
• FileModifyDate
• DateAcquired

The script in question

#!/bin/bash

shopt -s globstar

folder_name=/home/user/Pictures
start_range=20170101
end_range=20180130


for filename in $folder_name/**/*; do

	if [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -DateTimeOriginal "$filename") =~ ^[0-9]+$ ]]; then
		DateTimeOriginal=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -DateTimeOriginal "$filename")
	        if  [ "$DateTimeOriginal" -gt $start_range ] && [ "$DateTimeOriginal" -lt $end_range ]; then
			/usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$DateTimeOriginal/' '-FileName=%f%-c.%e' "$filename"
			echo "Found a value"
		echo "Okay its $(tput setab 22)DateTimeOriginal$(tput sgr0)"

		fi

        elif [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -CreateDate "$filename") =~ ^[0-9]+$ ]]; then
                CreateDate=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -CreateDate "$filename")
                if  [ "$CreateDate" -gt $start_range ] && [ "$CreateDate" -lt $end_range ]; then
                        /usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$CreateDate/' '-FileName=%f%-c.%e' "$filename"
                        echo "Found a value"
                echo "Okay its $(tput setab 27)CreateDate$(tput sgr0)"
                fi

        elif [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -FileModifyDate "$filename") =~ ^[0-9]+$ ]]; then
                FileModifyDate=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -FileModifyDate "$filename")
                if  [ "$FileModifyDate" -gt $start_range ] && [ "$FileModifyDate" -lt $end_range ]; then
                        /usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$FileModifyDate/' '-FileName=%f%-c.%e' "$filename"
                        echo "Found a value"
                echo "Okay its $(tput setab 202)FileModifyDate$(tput sgr0)"
                fi


        elif [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -DateAcquired "$filename") =~ ^[0-9]+$ ]]; then
                DateAcquired=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -DateAcquired "$filename")
                if  [ "$DateAcquired" -gt $start_range ] && [ "$DateAcquired" -lt $end_range ]; then
                        /usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$DateAcquired/' '-FileName=%f%-c.%e' "$filename"
                        echo "Found a value"
                echo "Okay its $(tput setab 172)DateAcquired(tput sgr0)"
                fi

        elif [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -ModifyDate "$filename") =~ ^[0-9]+$ ]]; then
                ModifyDate=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -ModifyDate "$filename")
                if  [ "$ModifyDate" -gt $start_range ] && [ "$ModifyDate" -lt $end_range ]; then
                        /usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$ModifyDate/' '-FileName=%f%-c.%e' "$filename"
                        echo "Found a value"
                echo "Okay its $(tput setab 135)ModifyDate(tput sgr0)"
                fi

        else
                echo "No EXIF field found"

done

Things I have tried

1. Reducing the number of if calls

But it didn't much improve the execution time (maybe a few ms?). The syntax looks way less readable but what I did, was to add a lot of or ( || ) in the syntax to reduce to a single if call. It's not finished, I just gave it a test drive with 2 EXIF fields (DateTimeOriginal and CreateDate) to see if it could somehow improve time. But meeeh :/.

#!/bin/bash

shopt -s globstar

folder_name=/home/user/Pictures
start_range=20170101
end_range=20201230

for filename in $folder_name/**/*; do

        if [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -DateTimeOriginal "$filename") =~ ^[0-9]+$ ]] || [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -CreateDate "$filename") =~ ^[0-9]+$ ]]; then
                DateTimeOriginal=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -DateTimeOriginal "$filename")
		CreateDate=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -CreateDate "$filename")
                if  [ "$DateTimeOriginal" -gt $start_range ] && [ "$DateTimeOriginal" -lt $end_range ] || [ "$CreateDate" -gt $start_range ] && [ "$CreateDate" -lt $end_range ]; then
                        /usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$DateTimeOriginal/' '-directory<$CreateDate/' '-FileName=%f%-c.%e' "$filename"
                        echo "Found a value"
                echo "Okay its $(tput setab 22)DateTimeOriginal$(tput sgr0)"

                else
			echo "FINISH YOUR SYNTAX !!"
		fi

	fi
done

2. Playing around with find

To recursively find my image files in all my folders I first tried the find function, but that gave me a lot of headaches... When my image file name had some spaces in it, it just broke the image path strangely... And all answers I found on the web were gibberish, and I couldn't make it work in my script properly... Lost over 4 yours only on that specific issue !

To overcome the hurdle someone suggest to use shopt -s globstar with for filename in $folder_name/**/* and this works perfectly. But I have no idea If this could be the culprit of slow execution time?

3. Changing all [ ] into [[ ]]

That also didn't do the trick.

How to Improve the processing time ?

I have no Idea if it's related to my script or the exiftool call that makes the script so slow. This isn't that much of a complicated script, I mean, it's a comparison between 2 integers not a hashing of complex numbers.

I hope someone could guide me in the right direction :)

Thanks !

A choice selector in pure bash (and compatible with zsh), inspired by fzf which I'm working on

cross-posted from: https://lemm.ee/post/23155648

Here is the script.

#!/usr/bin/env bash
# Download and search youtube subs
# deps yt-dlp ,awk, perl, any one or more of either ugrep, ripgrep, grep
# usage "script youtube_url"


main() {
    url="$@"
    check_if_url
    get_video_id
    search_for_downloaded_matching_files
    set_download_boolean_flag
    download_subs
    read_and_format_transcript_file
    echo_description_file
    user_search
}


# Iterate over the array and add items to the new array if they match the regex
check_if_url() {
    local regex='^https://[^[:space:]]+$'
        if ! [[ $url =~ $regex ]]; then
            echo "Invalid input. Valid input is a url matching regex ${regex}"
            exit 1
        fi
}


get_video_id() {
    video_id=$(echo "$url" | sed -n 's/.*v=\([^&]*\).*/\1/p')
}


search_for_downloaded_matching_files() {
    # Find newest created files matching the video_id
    transcript_file="$(  /usr/bin/ls -t --time=creation "$PWD"/*${video_id}*\.vtt 2>/dev/null | head -n 1  )"
    description_file="$(  /usr/bin/ls -t --time=creation "$PWD"/*${video_id}*\.description 2>/dev/null | head -n 1  )"
}


set_download_boolean_flag() {
    if [ -n "$transcript_file" ] && [ -n "$description_file" ]; then
        download=0 # FALSE
    else
        download=1 # TRUE
    fi
}


download_subs() {
    if [ "$download" -eq 1 ]; then
        yt-dlp --restrict-filenames --write-auto-sub --skip-download "${url}"
        yt-dlp --restrict-filenames --sub-langs=eng --write-subs --skip-download "${url}"
        yt-dlp --restrict-filenames --write-description --skip-download "${url}"
        # Search files again since they were just downloaded
        search_for_downloaded_matching_files
    fi
}


read_and_format_transcript_file() {
    perl_removed_dupes="$(perl -0777 -pe 's/^\d\d.*\n.*\n.*<\/c>//gm' <"${transcript_file}")"
    local prefix="https://www.youtube.com/watch?v=${video_id}&t="
    local suffix="s"
    formated_transcript_file="$(awk -v pre="$prefix" -v suf="$suffix" '
    /^([0-9]{2}:){2}[0-9]{2}\.[0-9]{3}/ {
        split($1, a, /[:.]/);
        $1 = pre (int(a[1]*3600 + a[2]*60 + a[3]) - 3) suf;
        sub(/ --> [0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}/, "");
        sub(/ align:start position:0%$/, "");
        print;
        next;
    }
    {
        sub(/ align:start position:0%$/, "");
        print;
    }
    ' <<<"${perl_removed_dupes}")"
    #CRLF for ugrep to avoid ?bug? where before lines are not all outputted
    formated_transcript_file_CRLF=$(printf '%b' "$formated_transcript_file" | sed 's/$/\r/')
}


echo_description_file() {
    cat "${description_file}"
}


user_search() {
    echo -e "\n\n"
    read -rp "Enter regex (read as raw input): " search_term

    : ${app_count:=0}

    if command -v ug >/dev/null 2>&1; then
        echo -e "\n\n\n\n"
        echo "Ugrep output"
        ug --pretty=never -B2 -A1 -i -Z+-~1 -e "${search_term}" --andnot "^https?:\/\/"  <<<"$formated_transcript_file_CRLF"
        ((app_count++))
    fi

    if command -v rg >/dev/null 2>&1; then
        echo -e "\n\n\n\n"
        echo "Ripgrep output"
        rg -iP -B2 -A7 "^(?!https?:\/\/).*\K${search_term}" <<<"$formated_transcript_file"
        ((app_count++))
    fi
    
    if [ "$app_count" -eq 0 ]; then
        echo -e "\n\n\n\n"
        echo "Grep output"
        grep -iP -B2 -A1 "${search_term}" <<<"$formated_transcript_file"
        echo -e "\n\n"
        echo "Consider installing ripgrep and ugrep for better search"
        ((app_count++))
    fi
}


main "$@"


    

I made a script that downloads from youtube super fast using a custom aria2 build.

Aria2 https://github.com/P3TERX/Aria2-Pro-Core/releases

ffmpeg build https://github.com/yt-dlp/FFmpeg-Builds/releases

I choose ffmpeg-master-latest-linux64-gpl.tar.xz

#!/usr/bin/env bash
#set -x

if [[ -z $@ ]]; then
    echo "specify download url"
    exit
fi

dir_dl="$PWD"
url="$@"

ffmpeg_dir="$HOME/.local/bin.notpath/"
download_archive_dir="$HOME/Videos/yt-dlp/"
download_archive_filename=".yt-dlp-archived-done.txt"

mkdir -p "$download_archive_dir"

youtube_match_regex='^.*(youtube[.]com|youtu[.]be|youtube-nocookie[.]com).*$'

if [[ "$1" =~ $youtube_match_regex ]]; then
    url="$(echo "$@" | perl -pe 's/((?:http:|https:)*?\/\/(?:www\.|)(?:youtube\.com|m\.youtube\.com|youtu\.|#youtube-nocookie\.com).*(?:c(?:hannel)?\/|u(?:ser)?\/|v=|v%3D|v\/|(?:a|p)\/(?:a|u)\/\d.*\/|watch\?|vi(?:=|\/)|\/#embed\/|oembed\?|be\/|e\/)([^&?%#\/\n]+)).*/$1/gm')"
    yt-dlp \
    --check-formats \
    --clean-info-json \
    --download-archive "$download_archive_dir$download_archive_filename" \
    --embed-chapters \
    --embed-info-json \
    --embed-metadata \
    --embed-thumbnail \
    --external-downloader aria2c \
    --downloader-args \
    "aria2c: \
        --allow-piece-length-change=true \
        --check-certificate=false \
        --console-log-level=notice \
        --content-disposition-default-utf8=true \
        --continue=true \
        --disk-cache=8192 \
        --download-result=full \
        --enable-mmap \
        --file-allocation=falloc \
        --lowest-speed-limit=100K \
        --max-concurrent-downloads=16 \
        --max-connection-per-server=64 \
        --max-mmap-limit=8192M \
        --max-resume-failure-tries=5 \
        --max-file-not-found=2 \
        --max-tries=3 \
        --min-split-size=64K \
        --no-file-allocation-limit=8192M \
        --piece-length=64k \
        --realtime-chunk-checksum=false \
        --retry-on-400=true \
        --retry-on-403=true \
        --retry-on-406=true \
        --retry-on-unknown=true \
        --retry-wait=1 \
        --split=32 \
        --stream-piece-selector=geom \
        --summary-interval=0 " \
    --ffmpeg-location "$ffmpeg_dir" \
    --output "$dir_dl"'/%(channel)s/%(title)s_%(channel)s_%(upload_date>%Y-%m-%d)s_%(duration>%H-%M-%S)s_%(resolution)s.%(ext)s' \
    --prefer-free-formats \
    --remux-video mkv \
    --restrict-filenames \
    --sponsorblock-remove "filler,interaction,intro,music_offtopic,outro,preview,selfpromo,sponsor" \
    --sub-langs "en.*,live_chat" \
    --write-auto-subs \
    --write-description \
    --write-info-json \
    --write-playlist-metafiles \
    --write-subs \
    --write-thumbnail \
    "$url"
else
    yt-dlp \
    --download-archive "$download_archive_dir$download_archive_filename" \
    --embed-chapters \
    --ffmpeg-location "$ffmpeg_dir" \
    --http-chunk-size 10M \
    --output "$dir_dl/%(title)s_%(duration>%H-%M-%S)s_%(upload_date>%Y-%m-%d)s_%(resolution)s_URL_(%(id)s).%(ext)s" \
    --prefer-free-formats \
    --restrict-filenames \
    "$url"
fi

[SOLVED] Solution: https://lemmy.ml/comment/4317564

I am doing all of this using ChatGPT, I know enough bash to understand the script partially, but not enough to write the script myself.

I recently posted bash script(click) to download songs on spotify using Spotdl Updated Post

This is all good, but I am not trying to download whole playlists and I want to make sure to

1. Not download any songs multiple times by comparing the files you are trying to download with the songs you have downloaded already.
2. Add the songs' URL to the Archieve_file so it doesn't download it again. i.e., compare and if present, exit loop, if not present in file, download and add link to the file.

This was easier when I was dealing with only song links and not playlist links. But now, playlists complicates the equation, but if I can achieve this, I can basically add this script to crontab and make sure I always have a local copy of the songs in my playlist and these playlists would be checked regularly for new downloads and new songs would be downloaded. This is really cool!

Now, the complications I am facing, I don't know how to get the URLs of the spotify songs out of spotify playlists using spotdl or any cli package. If I can do this then, I can make the script go through each link clean the link of unncessary attributes and then download it or if it's downloaded already, move on to the next song.

Now, I don't know how to do it. It would be very helpful if you guys could share any scripts that you have which will achieve this effect or help me get the song URL's from a playlist using a cli package.

PS: I might need some time to reply, I might sleep rn Thank you for your help! ___

I would rather prefer that you would buy spotify premium if you can. But, till you can, there's always spotdl which can allow you to download your songs

Installation of spotdl (github page click here): https://i.imgur.com/5g6uUgD.png

https://paste.debian.net/plain/1293528 a very simple script, you don't actually need the script, but it makes it easier to download songs.

btw, I am using file manager nemo here, if you want something else, change it to your default file manager, if you don't want to open folder, well remove last but one line.

execute this command first and when nano text edit appears, paste the debian pastebin in there.

This will remove all sponsors, download subtitles and view it when you are watching a video. Modify and Share this all over if you like! Edit: Give credits to this community or Lemmy in general if you are posting this or a modified form (please do share) of this elsewhere. Some popularity would do Lemmy good.

Few requirements:

1. You should be using Linux to run it.
2. Create a folder named yt-dlp in Videos folder or else change the location in the script below.
3. You should have yt-dlp not youtube-dl on your system. You can do this by sudo apt install yt-dlp or use your distro's package installer.
4. Save this file with any name in your home folder (or whichever folder you are you most comfortable with and give it execution permissions by chmod +x name

Debian Pastebin Thanks to folds at debian for making a tor network friendly pastebin. I have noticed that sometimes scripts get corrupted here, so best copy this from pastebin: https://paste.debian.net/1293211/

edit: deleted the codeblock as it was not rendering properly https://i.imgur.com/1lrTcdT.png

You can get the updated code here https://paste.debian.net/1293211/ Just paste it on to your notepad and give it execution permissions.

#edit 1: The program can be improved, if you guys feel like an improvement is needed, copy the entire program modify the parts you think can be made better and paste it in the comments or paste the pastebin link in the comments. I realize there are applications which are gui for yt-dlp but I felt they lacked many options. You can go on the man page, and learn and add attributes to make this script better so that it better suits your needs.

#man page aka github page: yt-dlp

A lot of people haven't heard of the at command, and I just figured I'd post up a little bit about it. I use it as a semi-alarm clock type reminder.

Structure of command the way I use it:

echo "notify-send -u critical -t 0 'YOUR URGENT MESSAGE'" | at "2pm Jun 18"

I actually wrote a tiny function so that I don't have to remember the -u critical -t 0 part. That is to keep the notification box from timing out. Now tomorrow (June 18th) at 2pm, a message box will pop up with the text "YOUR URGENT MESSAGE".

There are options for running commands (like cron, but you don't have to edit, then delete your edit after running). It is great if you don't want to mess with cron, or another utility for a quick reminder.

My quick little function looks like this:

notify_me ()
{
    if [[ $# -ne 2 ]]; then
        echo 'Usage:  notify_me "message" time/date' 1>&2;
        echo '                   make sure to enclose message in quotes' 1>&2;
        echo "                   also enclose date if it's more than just a time" 1>&2;
        return;
    fi;
    echo "notify-send -u critical -t 0 '$1'" | at "$2"
}

So say I have to remind myself to call the wife when the work day is over. I'll do something like:

notify_me "Check with the ol' ball and chain" "4:45pm"

I'm sure if you wanted to, you could make a very quick zenity (or whatever you like) gui for it, if that's the way you roll. Now I'll get a nice little pop-up 15 minutes before I check out. Tiny little things like this are why I enjoy the command line so much.

For more uses, check out the man page, or the tldr. It's not complicated and I find it useful.

Since the DNS lapsed for the wiki, someone grabbed a copy and put it up on github.

There is also the original webserver IP for /etc/hosts use, as the server is still there.

It's basically a builtin that you compile that can interface with routines in shared libraries, which means you could make GTK programs, network stacks, and more, all with bash and no external tools!

HI,

I don't use bar and status, only WM and some apps, so I use PS1 for see clock and battery status, clock change each time I enter a new prompt line but status battery change only with new shell

example :

PS1="$(cat /sys/class/power_supply/BAT0/capacity)% \A"

thanks

linux terminal

"We want to document one-liners for frequent (non-trivial) tasks executed in the shell"

linux shell manpage help

I wasn't easily able to find a command line study timer so I made one.

I'm not a great programmer so I was wondering if I could get some feedback. Cheers!

A Static Site Generator Made by Taminaru in Bash!

The current features are:

• Creating and editing posts in Markdown.

• Using templates, variables.

• Configuration, both local and universal (explained in README)

It's around 80% complete, so any contribution is appreciated (i would help if i wasn't lazy af lul).

Anyway here's the source code: https://codeberg.org/taminaru/BSG