{"id":1026,"date":"2018-02-21T13:00:25","date_gmt":"2018-02-21T13:00:25","guid":{"rendered":"http:\/\/chewett.co.uk\/blog\/?p=1026"},"modified":"2018-02-21T21:11:00","modified_gmt":"2018-02-21T21:11:00","slug":"cloning-multiple-git-repositories-remote-host-clone-repos","status":"publish","type":"post","link":"https:\/\/chewett.co.uk\/blog\/1026\/cloning-multiple-git-repositories-remote-host-clone-repos\/","title":{"rendered":"Cloning multiple git repositories from a remote host with clone-all-the-repos"},"content":{"rendered":"

This post goes into details about what the clone-all-the-repos\u00a0GitHub\u00a0repository<\/a> does and why it does it.<\/p>\n

<\/p>\n

What is clone-all-the-repos GitHub repository<\/h2>\n

I have a number of git repositories on a couple of servers that I want to clone at once. Manually doing this each time I set up a development environment\u00a0is tedious so I wanted to script it up.<\/p>\n

Originally there was a quick script written for this but when using it recently I decided to clean it up and publish it.<\/p>\n

The script follows the process:<\/p>\n

    \n
  1. Recursively loop through directories to find git repos (This is found using guessing if something is a git repository based on known files and folders)<\/li>\n
  2. Create “git clone” commands, that can be used by the remote machine<\/li>\n
  3. Create “mr\u00a0register” commands, if that option has been provided<\/li>\n<\/ol>\n

    The newly cleaned script is now available on GitHub under the project clone-all-the-repos<\/a>.<\/p>\n

    Using clone-all-the-repos<\/h2>\n

    A basic example of the command is:<\/p>\n

    python get_clone_script.py ~\/Documents\/git\/ hostname --mr-register<\/code><\/pre>\n
    The first argument to the script is the root directory that the script will search from. From here it will search all subdirectories to see if It can find any git repositories.<\/p>\n
    The second argument of the script is the hostname. This will be used when constructing the clone command. The hostname is used to create a clone command such as<\/p>\n
    git clone foobar:~\/Documents\/git\/repo1.git repo1.git<\/code><\/pre>\n
    Here foobar<\/code> is the value we passed to the script as the hostname parameter. Depending on your git setup this can be a fully qualified domain name, such as hostname.foobar.com<\/code>, or a ssh alias such as foobarhost<\/code>.<\/p>\n
    Finally I tell the script I want to output the mr register commands following the clone by passing --mr-register<\/code>.\u00a0If this isn’t\u00a0provided the script will not print out the mr register commands.<\/p>\n
    A final optional argument is --git-clone-command<\/code>\u00a0which allows you to change what the script uses in place of git clone. This could be used to add additional flags to git clone, or adding a custom cloning command.<\/p>\n
    Future Improvements<\/h2>\n
    One of the improvements I am going to add in the future is\u00a0to add some better logic for finding and checking repositories.<\/p>\n
    The current logic for finding repositories is probably sufficient\u00a0but I could add further logic to run a git command and check\u00a0to see if it identifies it as a git repository.<\/p>\n","protected":false},"excerpt":{"rendered":"
    This post goes into details about what the clone-all-the-repos\u00a0GitHub\u00a0repository does and why it does it.<\/p>\n","protected":false},"author":1,"featured_media":1028,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[98],"tags":[228,76,184],"class_list":["post-1026","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-software","tag-git","tag-github","tag-python"],"wppr_data":{"cwp_meta_box_check":"No"},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/clone_all_the_repos_github.jpg?fit=800%2C800&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p2toWX-gy","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":1626,"url":"https:\/\/chewett.co.uk\/blog\/1626\/create-clone-commands-for-all-repositories-on-a-github-page\/","url_meta":{"origin":1026,"position":0},"title":"Create Clone Commands for all Repositories on a GitHub page","author":"Chewett","date":"October 13, 2018","format":false,"excerpt":"This post talks about how you can create git clone links for all the repositories from a GitHub page. Creating a list of clone commands from a list of Repositories To run this code to create the list of commands you need to open your browsers console\u00a0window. On Chrome you\u2026","rel":"","context":"In "Informational"","block_context":{"text":"Informational","link":"https:\/\/chewett.co.uk\/blog\/category\/informational\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/10\/download_all_github_repos-1.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/10\/download_all_github_repos-1.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/10\/download_all_github_repos-1.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/10\/download_all_github_repos-1.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":519,"url":"https:\/\/chewett.co.uk\/blog\/519\/configuring-teamcity-php-project\/","url_meta":{"origin":1026,"position":1},"title":"Configuring TeamCity for a PHP project","author":"Chewett","date":"September 16, 2017","format":false,"excerpt":"This post describes configuring TeamCity for a PHP project using composer. Setting up continuous integration of unit testing from a git repository. Setting up PHP, composer and git The first step to running your PHP project on TeamCity is installing PHP and the appropriate\u00a0libraries. The only requirement for this is\u2026","rel":"","context":"In "Informational"","block_context":{"text":"Informational","link":"https:\/\/chewett.co.uk\/blog\/category\/informational\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/09\/configuring_php_unit_testing_with_teamcity.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/09\/configuring_php_unit_testing_with_teamcity.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/09\/configuring_php_unit_testing_with_teamcity.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/09\/configuring_php_unit_testing_with_teamcity.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1839,"url":"https:\/\/chewett.co.uk\/blog\/1839\/raspberry-pi-cluster-node-08-slave-helper-functions\/","url_meta":{"origin":1026,"position":2},"title":"Raspberry Pi Cluster Node – 08 Slave Helper Functions","author":"Chewett","date":"December 19, 2018","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by adding a number of slave helper functions.\u00a0 This update will begin the process of fully automating the slaves. Preparing the Slaves for Automation Before the slaves are ready to be fully automated there are a number of commands\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":2827,"url":"https:\/\/chewett.co.uk\/blog\/2827\/the-risks-of-windows-and-linux-line-endings-in-git-with-pogoapi\/","url_meta":{"origin":1026,"position":3},"title":"The Risks of Windows and Linux Line endings in Git with PoGoAPI","author":"Chewett","date":"March 6, 2021","format":false,"excerpt":"This talks about the minor changes needed for the api_hashes api to allow verifying API's. Original Problem Someone contacted me and said they couldnt verify the API's against the hashes I provided. I looked into this and locally there didnt seem to be an issue as the API's matched directly.\u2026","rel":"","context":"In "Informational"","block_context":{"text":"Informational","link":"https:\/\/chewett.co.uk\/blog\/category\/informational\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/03\/pogoapi_git_issues_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/03\/pogoapi_git_issues_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/03\/pogoapi_git_issues_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/03\/pogoapi_git_issues_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/03\/pogoapi_git_issues_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":453,"url":"https:\/\/chewett.co.uk\/blog\/453\/automatic-installation-commonly-used-packages\/","url_meta":{"origin":1026,"position":4},"title":"Automatic installation of commonly used packages","author":"Chewett","date":"July 15, 2017","format":false,"excerpt":"A number of years ago I was introduced to using Linux day to day. During this period of time I ran a large number of virtual machines to test its performance and various aspects which make it \"linux\". Typically when I first set up a system I would install a\u2026","rel":"","context":"In "Informational"","block_context":{"text":"Informational","link":"https:\/\/chewett.co.uk\/blog\/category\/informational\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":343,"url":"https:\/\/chewett.co.uk\/blog\/343\/raspberrypivcgencmd-python-library-access-raspberry-pi-vcgencmd-command\/","url_meta":{"origin":1026,"position":5},"title":"RaspberryPiVcgencmd A python library to access Raspberry Pi vcgencmd command","author":"Chewett","date":"May 31, 2017","format":false,"excerpt":"To access some of the lower level aspects of the Raspberry Pi's config you can use the built in command vcgencmd. This is a useful on the command line but to easily access it in a program you need to parse the output. I am planning to use this command\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/05\/vcgencmd.png?fit=628%2C288&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/05\/vcgencmd.png?fit=628%2C288&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/05\/vcgencmd.png?fit=628%2C288&ssl=1&resize=525%2C300 1.5x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/1026","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=1026"}],"version-history":[{"count":1,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/1026\/revisions"}],"predecessor-version":[{"id":1027,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/1026\/revisions\/1027"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media\/1028"}],"wp:attachment":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=1026"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=1026"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=1026"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}