Apt Get本地资源库搭建

将hortonworks镜像服务器上的 Ambari, HDP, HDP-UTILS 仓库同步到本地,用于在网络环境不佳状态下搭建Hadoop集群。


参考


Apt-mirror

Apt-mirror是一个为Debian和Ubuntu提供任何部分镜像的工具。

安装apt-mirror:$ sudo apt-get install apt-mirror

修改配置:

$ sudo vim /etc/apt/mirror.list

############# config ##################
#
# set base_path    /var/spool/apt-mirror
#
# set mirror_path  $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch  <running host architecture>
# set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads     20
set _tilde 0
############# end config ##############

deb http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu14 HDP-UTILS main
deb http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.1.2 Ambari main
deb http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.3.2.0 HDP main
clean deb http://public-repo-1.hortonworks.com

其中 deb 开头的三行, 是要同步的远端服务器上的三个目录:

更新

每次更新都可以执行下面的命令, 命令自动退出以后, 查看是否有 error, 如果没有, 表示更新完成. 同步的内容不会再变动, 除非再次执行这条命令.

$ su root
$ su - apt-mirror -c apt-mirror

更新后在/etc/apt/mirror.list中添加的源就会被同步到/var/spool/apt-mirror这个路径下,各个文件夹具体的含义如下:

/var/spool/apt-mirror/mirror
   Mirror places here

/var/spool/apt-mirror/skel
   Place for temporarily downloaded indexes

/var/spool/apt-mirror/var
   Log files placed here. URLs and MD5 checksums also here.

使用镜像原

从本地访问

如果源已经被下载到本地,可以直接在/etc/apt/source.list中编辑写入源地址

$ sudo vim /etc/apt/source.list

deb file:///var/www/html/hdp/ambari/ubuntu14/2.x/updates/2.1.2 Ambari main
deb file:///var/www/html/hdp/HDP-UTILS-1.1.0.20/repos/ubuntu14 HDP-UTILS main
deb file:///var/www/html/hdp/HDP/ubuntu14/2.x/updates/2.3.2.0 HDP main

供web访问

如果需要通过web来访问,首先需要一个web服务器如Apache

$ apt-get install apache

Apache服务安装后直接可以运行,打开浏览器,在地址栏输入:localhost或者http://127.0.0.1,看到It works,表示安装成功!默认根目录:/var/www/

将mirror镜像化资源库的文件夹软连接到apache目录下:

$ ln -s /var/spool/apt-mirror/mirror/public-repo-1.hortonworks.com/ /var/www/html/hdp

用浏览器访问 http://host-ip/hdp 来查看是否能出现文件列表, 出现的话, 说明可以通过web访问.

然后重新更新apt-get的sources.list文件

$ sudo vim /etc/apt/source.list

deb http://cci-ambari/hdp/ambari/ubuntu14/2.x/updates/2.1.2 Ambari main
deb http://cci-ambari/hdp/HDP-UTILS-1.1.0.20/repos/ubuntu14 HDP-UTILS main
deb http://cci-ambari/hdp/HDP/ubuntu14/2.x/updates/2.3.2.0 HDP main

然后重新做apt-get update来完成更新。


Aptly

Aptly也是一个利用快照来做镜像的工具,目标是为Debian wheezy(Ubuntu 通用)创建整合了安全更新的镜像,使用快照在各个hosts上安装相同的package。

准备

安装Aptly,用Aptly的GPG key来访问公用资源库,如果没有GPG key,可以创建一个:

$ gpg --gen-key
gpg (GnuPG) 1.4.12; Copyright (C) 2012 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection?
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: Andrey Smirnov
Email address: me@smira.ru
Comment: Signing repos
You selected this USER-ID:
    "Andrey Smirnov (Signing repos) <me@smira.ru>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/6430156A 2014-10-09
      Key fingerprint = 52EC DF0E EE8B 5DDF CC8F  7038 B9C8 F8B4 6430 156A
uid                  Andrey Smirnov (Signing repos) <me@smira.ru>
sub   2048R/E0C10001 2014-10-09

创建镜像

假设要安装的镜像是wheezy Debian,架构是amd64,创建主组件:

$ aptly mirror create -architectures=amd64 -filter='Priority (required) | Priority (important) | Priority (standard)' wheezy-main http://ftp.ru.debian.org/debian/ wheezy main

Looks like your keyring with trusted keys is empty. You might consider importing some keys.
If you're running Debian or Ubuntu, it's a good idea to import current archive keys by running:

  gpg --no-default-keyring --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export | gpg --no-default-keyring --keyring trustedkeys.gpg --import

(for Ubuntu, use /usr/share/keyrings/ubuntu-archive-keyring.gpg)

Downloading http://ftp.ru.debian.org/debian/dists/wheezy/InRelease...
Downloading http://ftp.ru.debian.org/debian/dists/wheezy/Release...
Downloading http://ftp.ru.debian.org/debian/dists/wheezy/Release.gpg...
gpgv: Signature made Sat 12 Jul 2014 10:59:56 AM UTC using RSA key ID 46925553
gpgv: Can't check signature: public key not found
gpgv: Signature made Sat 12 Jul 2014 11:04:06 AM UTC using RSA key ID 65FFB764
gpgv: Can't check signature: public key not found

Looks like some keys are missing in your trusted keyring, you may consider importing them from keyserver:

gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 46925553 65FFB764

Sometimes keys are stored in repository root in file named Release.key, to import such key:

wget -O - https://some.repo/repository/Release.key | gpg --no-default-keyring --keyring trustedkeys.gpg --import

ERROR: unable to fetch mirror: verification of detached signature failed: exit status 2

报了个错说missing keys in trusted keyring,可以导入默认的Debian keyring:

$ gpg --no-default-keyring --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export | gpg --no-default-keyring --keyring trustedkeys.gpg --import
gpg: key 6430156A: public key "Andrey Smirnov (Signing repos) <me@smira.ru>" imported
gpg: key 2A194991: public key "Andrey Smirnov <me@smira.ru>" imported
gpg: key B98321F9: public key "Squeeze Stable Release Key <debian-release@lists.debian.org>" imported
gpg: key 473041FA: public key "Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster@debian.org>" imported
gpg: key 65FFB764: public key "Wheezy Stable Release Key <debian-release@lists.debian.org>" imported
gpg: key 46925553: public key "Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>" imported
gpg: Total number processed: 6
gpg:               imported: 6  (RSA: 6)
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u

重新创建:

$ aptly mirror create -architectures=amd64 -filter='Priority (required) | Priority (important) | Priority (standard) | nginx | postgresql | redis-server | memcached | ruby | golang' -filter-with-deps wheezy-main http://ftp.ru.debian.org/debian/ wheezy main
Downloading http://ftp.ru.debian.org/debian/dists/wheezy/InRelease...
Downloading http://ftp.ru.debian.org/debian/dists/wheezy/Release...
Downloading http://ftp.ru.debian.org/debian/dists/wheezy/Release.gpg...
gpgv: Signature made Sat 12 Jul 2014 10:59:56 AM UTC using RSA key ID 46925553
gpgv: Good signature from "Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>"
gpgv: Signature made Sat 12 Jul 2014 11:04:06 AM UTC using RSA key ID 65FFB764
gpgv: Good signature from "Wheezy Stable Release Key <debian-release@lists.debian.org>"

Mirror [wheezy-main]: http://ftp.ru.debian.org/debian/ wheezy successfully added.
You can run 'aptly mirror update wheezy-main' to download repository contents.

现在签名已经被验证了。标签-filter=允许我们自定义下载的packages的数目,在第一部分中,Priority (required) | Priority (important) | Priority (standard)是Debian系统基础的需求,然后一些其他的package也可以被添加进去,如nginx, postgresql-filter-with-deps标志指定了aptly需要包含哪些package依赖。

创建 wheezy-updateswheezy-security镜像:

$ aptly mirror create -architectures=amd64 -filter='Priority (required) | Priority (important) | Priority (standard) | nginx | postgresql | redis-server | memcached | ruby | golang' -filter-with-deps wheezy-updates http://ftp.ru.debian.org/debian/ wheezy-updates main
...

$ aptly mirror create -architectures=amd64 -filter='Priority (required) | Priority (important) | Priority (standard) | nginx | postgresql | redis-server | memcached | ruby | golang' -filter-with-deps wheezy-security http://security.debian.org/ wheezy/updates main
...

###更新镜像

$ aptly mirror update wheezy-main
Downloading http://ftp.ru.debian.org/debian/dists/wheezy/InRelease...
Downloading http://ftp.ru.debian.org/debian/dists/wheezy/Release...
Downloading http://ftp.ru.debian.org/debian/dists/wheezy/Release.gpg...
gpgv: Signature made Sat 12 Jul 2014 10:59:56 AM UTC using RSA key ID 46925553
gpgv: Good signature from "Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>"
gpgv: Signature made Sat 12 Jul 2014 11:04:06 AM UTC using RSA key ID 65FFB764
gpgv: Good signature from "Wheezy Stable Release Key <debian-release@lists.debian.org>"
Downloading & parsing package files...
Downloading http://ftp.ru.debian.org/debian/dists/wheezy/main/binary-amd64/Packages.bz2...
Applying filter...
Packages filtered: 35933 -> 304.
Building download queue...
Download queue: 304 items (147.18 MiB)
Downloading http://ftp.ru.debian.org/debian/pool/main/g/gnutls26/libgnutls26_2.12.20-8+deb7u2_amd64.deb...
Downloading http://ftp.ru.debian.org/debian/pool/main/e/eglibc/locales_2.13-38+deb7u2_all.deb...
Downloading http://ftp.ru.debian.org/debian/pool/main/p/perl/perl-modules_5.14.2-21+deb7u1_all.deb...
Downloading http://ftp.ru.debian.org/debian/pool/main/v/vim/vim-tiny_7.3.547-7_amd64.deb...
Downloading http://ftp.ru.debian.org/debian/pool/main/libt/libtext-iconv-perl/libtext-iconv-perl_1.7-5_amd64.deb...
....

Mirror `wheezy-main` has been successfully updated.

Package文件被下载到了~/.aptly/pool/

实践

在现有的开发环境中,我们已经有同事在10.10.82.15上已经创建好适用于Debian/Ubuntu的HDP和HDP镜像,在使用时候,只需要在本地/etc/apt/sources.list中添加新的repo地址,如deb [arch=amd64] http://10.10.82.15:8080 HDP main, 其中[arch=amd64]标致必须添加,表示这个repo的结构是64位的。

然后将Aptly-key复制到本地:

$ scp root@10.10.82.15:/script/aptly-key.pub /tmp

最后添加复制来的Aptly-key,并尝试更新apt-get以检查正确性:

$ apt-key add /tmp/aptly-key.pub
$ apt-get update
Kaka Chen /
Published under (CC) BY-NC-SA in categories Linux  tagged with Linux  Apt