You are not logged in.

#1 2016-10-10 18:40:57

xse
Member
From: 127.0.0.1
Registered: 2016-10-10
Posts: 5
Website

simple script to clone repository

Hello there ! I made a litlle shell script to clone the main repository using wget as rsync can't be used.

#!/usr/bin/env sh
# First launch : download repo from https://packages.pacbsd.org/ to /srv/http/packages.pacbsd.org and delete all html files
# don't forget to chmod http:http on /srv/http/packages.pacbsd.org for nginx.
#
# Second launch and after : Update the repo & delete all html files

sudo -u http wget -N -P "/srv/http/" --mirror "https://packages.pacbsd.org/"  # || printf "%s\n" "error wget" && exit 1

find /srv/http/packages.pacbsd.org/ -name "*.html" -delete # || printf "%s\n" "error removing html files" && exit 1

As you can see I don't handle errors, as I don't think wget always return a 0..

So i've set a mirror of the main repo here : http://pacbsd.krkrkr.org/ ( https version is comming tongue ). This mirror is hosted at OVH's datacenter, in Roubaix ( FR )
Here is the result of a speedtest on this server :

Selecting best server based on latency...
b'Hosted by Orange (Paris) [1.88 km]: 6.597 ms'
Testing download speed........................................
Download: 787.94 Mbit/s
Testing upload speed..................................................
Upload: 430.18 Mbit/s

And thats it ! if someone have any improvement or something like that, feel free to post it !

Last edited by xse (2016-10-10 19:31:44)

Offline

#2 2016-10-10 21:14:22

vendion
Administrator
From: Tennessee, USA
Registered: 2015-05-01
Posts: 21
Website

Re: simple script to clone repository

Thanks, your mirror has also been added to the mirrorlist updatable from https://github.com/PacBSD/abs/blob/mast … mirrorlist or from the package currently in [testing].

Offline

#3 2016-10-11 00:00:14

xse
Member
From: 127.0.0.1
Registered: 2016-10-10
Posts: 5
Website

Re: simple script to clone repository

Awesome ! https will be available as soon as possible smile

You can check the last update time (set to every 6hours now for testing) at http://krkrkr.org/ along with the last modified files.
If you think it's nice to have them (time & last modified files) somewhere else, just let me know. Thoses informations are also visible on their own page at http://krkrkr.org/pacbsd_mirror.html

The informations pages are updated by the same script that updates the repository, so they should always be up to date unless something goes wrong..
If an admin/develloper feels the need to have an ssh acces on the virtual machine that run it, let me know !

Offline

#4 2016-10-11 06:37:12

vendion
Administrator
From: Tennessee, USA
Registered: 2015-05-01
Posts: 21
Website

Re: simple script to clone repository

Can you write that information to a JSON file on the server somewhere?  That way I can make the site pull it and display it in the mirror status page.

Offline

#5 2016-10-11 12:19:50

xse
Member
From: 127.0.0.1
Registered: 2016-10-10
Posts: 5
Website

Re: simple script to clone repository

Ok i've done it here : http://pacbsd.krkrkr.org/mirror_info.json i'm not familliar at all with json so let me know if I did it wrong. I don't even know if that is a valid json file..

So the json file looks like that :

{"pacbsd_mirror": {
  "update_date": "2016-10-11",
  "udpate_hour": "23:32:06",
  "last_files": [
    {"file": "mirror_info.json", "date": "11/10/2016-23:31:05"},
    {"file": "testing/os/amd64/testing.files.tar.gz", "date": "11/10/2016-04:06:57"},
    {"file": "testing/os/amd64/testing.files", "date": "11/10/2016-04:06:57"},
    {"file": "testing/os/amd64/testing.db.tar.gz", "date": "11/10/2016-04:06:57"},
    {"file": "testing/os/amd64/testing.db", "date": "11/10/2016-04:06:57"},
    {"file": "testing/os/amd64/pacman-mirrorlist-20161010-1-amd64.pkg.tar.xz.sig", "date": "11/10/2016-04:05:26"},
    {"file": "testing/os/amd64/pacman-mirrorlist-20161010-1-amd64.pkg.tar.xz", "date": "11/10/2016-04:05:26"},
    {"file": "testing/os/amd64/testing.files.tar.gz.old", "date": "11/10/2016-03:45:59"},
    {"file": "testing/os/amd64/testing.db.tar.gz.old", "date": "11/10/2016-03:45:59"},
    {"file": "testing/os/amd64/pacbsd-keyring-20161010-1-any.pkg.tar.xz.sig", "date": "11/10/2016-03:03:27"},
    {"file": "testing/os/amd64/pacbsd-keyring-20161010-1-any.pkg.tar.xz", "date": "11/10/2016-03:03:27"},
    {"file": "pacbsd-keyring/pacbsd-keyring-20161010.tar.gz", "date": "11/10/2016-02:13:08"},
    {"file": "iso/PacBSD-amd64-20160721.iso.torrent", "date": "14/09/2016-03:25:59"},
    {"file": "iso/PacBSD-amd64-20160721.img.torrent", "date": "14/09/2016-03:25:59"},
    {"file": "iso/sha1sum.txt", "date": "04/09/2016-16:01:59"}
   ]
}}

Here is the whole ugly roll script i use to do it ( update mirror, update personnal page and create json file ).

#!/usr/bin/env zsh
sudo -u http wget -N -P "/srv/http/" --mirror "https://packages.pacbsd.org/"  # || printf "%s\n" "error wget" && exit 1
find /srv/http/packages.pacbsd.org/ -name "*.html" -delete # || printf "%s\n" "error removing html files" && exit 1
echo -n "" > /home/xse/www/krkrkr/_includes/pacbsd_update
DATE=`date +%Y-%m-%d`
HOUR=`date +%H:%M:%S`
LASTMODIF=`find /srv/http/packages.pacbsd.org -type f -printf "%-.22T+  %Tx %.8TX %p\n" | sort -r | head -15 | cut -f 2- -d ' ' | sed -r 's/\/srv\/http\/packages.pacbsd.org\///'`
echo -n "<br><p>Last update: ${DATE} ${HOUR}</p><br><span>Last modified files :</span><br><pre>${LASTMODIF}</pre>"  > /home/xse/www/krkrkr/_includes/pacbsd_update
chmod 777 /home/xse/www/krkrkr/_includes/pacbsd_update
chown xse:users /home/xse/www/krkrkr/_includes/pacbsd_update
cd /home/xse/www/krkrkr/ && sudo -u xse jekyll build --source /home/xse/www/krkrkr --destination /srv/http/krkrkr.org


LASTMODIFJSON=`find /srv/http/packages.pacbsd.org -type f -printf "%-.22T+  %Tx %.8TX %p\n" | sort -r | head -15 | cut -f 2- -d ' ' | sed -r 's/\/srv\/http\/packages.pacbsd.org\///' | while read -r line; do echo $line | awk '{printf "    {\"file\": \"%s\", \"date\": \"%s-%s\"},\n", $3, $1, $2}'; done | sed '$s/,\([^,]\)*$/\1/'`

echo -en "{\"pacbsd_mirror\": {\n  \"update_date\": \"${DATE}\",\n  \"udpate_hour\": \"${HOUR}\",\n  \"last_files\": [\n" > /srv/http/packages.pacbsd.org/mirror_info.json
echo ${LASTMODIFJSON} >> /srv/http/packages.pacbsd.org/mirror_info.json
echo -en "   ]\n}}\n" >> /srv/http/packages.pacbsd.org/mirror_info.json

I can increase the number of last modified files displayed if you think that is good to have the 50 last modified file for example.

Edit #1:: changed the script to improve the json file. I plan on using http://stedolan.github.io/jq/ to handle the json part in the next days

Last edited by xse (2016-10-11 18:05:22)

Offline

#6 2016-10-11 13:04:33

xse
Member
From: 127.0.0.1
Registered: 2016-10-10
Posts: 5
Website

Re: simple script to clone repository

Update : The HTTPS version is working great, i'll need to know if you want the mirror to be accessible by http too, or not, at the moment I redirect all http request to https like this in the nginx configuration ( i'm behind another nginx proxy that handle https with letsencrypt for me ) :

                if ( $http_x_forwarded_proto = 'http' ) {
                         return 301 https://$host$request_uri;
                }

So, let me know what you prefer ( I prefer it https only of course ).

Offline

#7 2016-10-12 09:12:17

vendion
Administrator
From: Tennessee, USA
Registered: 2015-05-01
Posts: 21
Website

Re: simple script to clone repository

If the mirror is available under http and https or https only is your call.  You tell us what protocols your mirror supports and we will go from there, as far as pacman goes we will probably have it go over  HTTPS.

As for your JSON that looks valid so we are good.

Offline

#8 2016-10-12 16:13:54

xse
Member
From: 127.0.0.1
Registered: 2016-10-10
Posts: 5
Website

Re: simple script to clone repository

Ok i prefer have it go over HTTPS only, it should be more "secure by default". ( If you think it's great to have http available too it's ok for me tho )
If you can update the mirrors page as well as the mirrorlist files with https instead of http smile

BTW: i've noticed that the mirrors page has a typo error, it sould be "Protocols" instead of "Protocals"
( I'm sorry for my english, i'm not a native english speaker sad )

Offline

Board footer

Powered by FluxBB