[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Fedora-livecd-list] Space savings discussed on fedora-packaging-list



Hi,

below discussion from Fedora Packaging FYI, in case the space saving
discussed there are of interest for you.

Original thread starts at:
https://www.redhat.com/archives/fedora-packaging/2007-April/msg00003.html

CU
thl

------- Original-Nachricht --------
Betreff: [Fedora-packaging] Hardlinking *.pyc and *.pyo
Datum: Tue, 3 Apr 2007 13:42:24 +0300
Von: Ville Skyttä <ville skytta iki fi>
Antwort an: Discussion of RPM packaging standards and practices for
Fedora Extras <fedora-packaging redhat com>
An: fedora-packaging Mailing List <fedora-packaging redhat com>

Hello,

Related to recent space saving discussions, I came across PLD's
rpm-build-macros package recently, and found that they hardlink
identical *.pyc and *.pyo.  In a lot of cases, they're the same,
and there's some potential for saving some MB "for free",
on my FC6 x86_64 box:

$ /usr/sbin/hardlink -ncv /usr/lib*/python2.4 2>&1 | tail -n 1
Would save 11116544

The PLD implementation looks like this:

# Hardlink binary identical .pyc and .pyo files
# (idea by glen <at> pld-linux <dot> org)
%__spec_install_post_py_hardlink {\
%{!?no_install_post_py_hardlink: __spec_install_post_py_hardlink() { \
[ ! -d "$RPM_BUILD_ROOT" ] || find "$RPM_BUILD_ROOT" -name '*.pyc' |
while read a; do \
        b="$(echo $a|sed -e 's/.pyc$/.pyo/')"; \
        if cmp -s "$a" "$b"; then \
                ln -f "$a" "$b"; \
        fi; \
done \
}; __spec_install_post_py_hardlink } }

The use of "cmp" would require diffutils installed.  Or the above
could be converted to use hardlink instead (which would have to be made
sure to be around) or maybe sha1sum (in coreutils, pretty much always
around in buildroots).

I suppose something like the above could be easily added to
redhat-rpm-config or rpm, eg. embedded in brp-python-bytecompile
or run after it in %__os_install_post.

Worth it?  Other comments?
-------- Original-Nachricht --------
Return-Path: <Thorsten_Leemhuis gmx de>
X-Original-To: rpmbuild localhost
Delivered-To: rpmbuild localhost thl home
Received: by truhe.thl.home (Postfix, from userid 500)	id 67F1B8110;
Tue,  3 Apr 2007 13:24:27 +0200 (CEST)
X-Original-To: thl localhost
Delivered-To: thl localhost thl home
Received: from truhe.thl.home (localhost [127.0.0.1])	by truhe.thl.home
(Postfix) with ESMTP id 510D280AD	for <thl localhost>; Tue,  3 Apr 2007
13:24:27 +0200 (CEST)
X-Original-To: fedora leemhuis info
Delivered-To: web550p1 basicbox7 server-home net
Received: from leemhuis.info [195.137.212.29]	by truhe.thl.home with
POP3 (fetchmail-6.3.6)	for <thl localhost> (single-drop); Tue, 03 Apr
2007 13:24:27 +0200 (CEST)
Received: from hormel.redhat.com (hormel.redhat.com [209.132.177.30])	by
basicbox7.server-home.net (Postfix) with ESMTP id 1B0C63741D9	for
<fedora leemhuis info>; Tue,  3 Apr 2007 13:17:29 +0200 (CEST)
Received: from listman.util.phx.redhat.com
(listman.util.phx.redhat.com [10.8.4.110])	by hormel.redhat.com
(Postfix) with ESMTP	id B188E72E40; Tue,  3 Apr 2007 07:17:30 -0400 (EDT)
Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com
[172.16.52.254])	by listman.util.phx.redhat.com (8.13.1/8.13.1) with
ESMTP id	l33BHSHq025579 for
<fedora-packaging listman util phx redhat com>;	Tue, 3 Apr 2007 07:17:28
-0400
Received: from mx1.redhat.com (mx1.redhat.com [172.16.48.31])	by
int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l33BHSBL023395	for
<fedora-packaging redhat com>; Tue, 3 Apr 2007 07:17:28 -0400
Received: from TZMXR01.htp-tel.de (tzmxr01.htp-tel.de [81.14.243.17])	by
mx1.redhat.com (8.13.1/8.13.1) with ESMTP id l33BHM8I029053	for
<fedora-packaging redhat com>; Tue, 3 Apr 2007 07:17:23 -0400
Received: from TZMXR01.htp-tel.de (localhost [127.0.0.1])	by
TZMXR01.htp-tel.de with ESMTP id l33BHGYK029546	for
<fedora-packaging redhat com>; Tue, 3 Apr 2007 13:17:16 +0200 (CEST)
Received: from truhe.thl.home (a81-14-184-169.net-htp.de
[81.14.184.169])	by TZMXR01.htp-tel.de with ESMTP id l33BHF0E029531	for
<fedora-packaging redhat com>; Tue, 3 Apr 2007 13:17:16 +0200 (CEST)
Received: from thl.ct.heise.de (localhost [127.0.0.1])	by truhe.thl.home
(Postfix) with ESMTP id 70DC580AD	for <fedora-packaging redhat com>;
Tue,  3 Apr 2007 13:17:15 +0200 (CEST)
Message-ID: <461237BB 5050700 leemhuis info>
Date: Tue, 03 Apr 2007 13:17:15 +0200
From: Thorsten Leemhuis <fedora leemhuis info>
User-Agent: Thunderbird 2.0b2 (X11/20070308)
MIME-Version: 1.0
To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging redhat com>
Subject: Re: [Fedora-packaging] Hardlinking *.pyc and *.pyo
References: <200704031342 25238 ville skytta iki fi>
In-Reply-To: <200704031342 25238 ville skytta iki fi>
X-Enigmail-Version: 0.94.3.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-RedHat-Spam-Score: 0
X-loop: fedora-packaging redhat com
X-BeenThere: fedora-packaging redhat com
X-Mailman-Version: 2.1.5
Precedence: junk
Reply-To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging redhat com>
List-Id: Discussion of RPM packaging standards and practices for
Fedora Extras	<fedora-packaging.redhat.com>
List-Unsubscribe:
<https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request redhat com?subject=unsubscribe>
List-Archive: <https://www.redhat.com/archives/fedora-packaging>
List-Post: <mailto:fedora-packaging redhat com>
List-Help: <mailto:fedora-packaging-request redhat com?subject=help>
List-Subscribe:
<https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request redhat com?subject=subscribe>
Sender: fedora-packaging-bounces redhat com
Errors-To: fedora-packaging-bounces redhat com
X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on
basicbox7.server-home.net
X-Spam-Level:
X-Spam-Status: No, score=-0.1 required=5.0 tests=AWL
autolearn=disabled 	version=3.0.3
X-UIDL: dg:!!WW-!!BIp"!23Y"!

On 03.04.2007 12:42, Ville Skyttä wrote:

> Worth it?  Other comments?

Hehe, sounds interesting. Did you tell the Live-CD guys about it? I'd
say they will probably very interested in something like this (they
could run hardlink directly for now until we decide what we want to do).

CU
thl


-------- Original-Nachricht --------
Return-Path: <Thorsten_Leemhuis gmx de>
X-Original-To: rpmbuild localhost
Delivered-To: rpmbuild localhost thl home
Received: by truhe.thl.home (Postfix, from userid 500)	id DB9EB8110;
Tue,  3 Apr 2007 13:54:57 +0200 (CEST)
X-Original-To: thl localhost
Delivered-To: thl localhost thl home
Received: from truhe.thl.home (localhost [127.0.0.1])	by truhe.thl.home
(Postfix) with ESMTP id C928480AD	for <thl localhost>; Tue,  3 Apr 2007
13:54:57 +0200 (CEST)
X-Original-To: fedora leemhuis info
Delivered-To: web550p1 basicbox7 server-home net
Received: from leemhuis.info [195.137.212.29]	by truhe.thl.home with
POP3 (fetchmail-6.3.6)	for <thl localhost> (single-drop); Tue, 03 Apr
2007 13:54:57 +0200 (CEST)
Received: from hormel.redhat.com (hormel.redhat.com [209.132.177.30])	by
basicbox7.server-home.net (Postfix) with ESMTP id 1EF8C374D37	for
<fedora leemhuis info>; Tue,  3 Apr 2007 13:45:41 +0200 (CEST)
Received: from listman.util.phx.redhat.com
(listman.util.phx.redhat.com [10.8.4.110])	by hormel.redhat.com
(Postfix) with ESMTP	id AAF3D73009; Tue,  3 Apr 2007 07:45:42 -0400 (EDT)
Received: from int-mx2.corp.redhat.com (nat-pool.util.phx.redhat.com
[10.8.5.200])	by listman.util.phx.redhat.com (8.13.1/8.13.1) with
ESMTP id	l33Bjfnb029939 for
<fedora-packaging listman util phx redhat com>;	Tue, 3 Apr 2007 07:45:41
-0400
Received: from mx2.redhat.com (mx2.redhat.com [10.255.15.25])	by
int-mx2.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l33Bje9H030402	for
<fedora-packaging redhat com>; Tue, 3 Apr 2007 07:45:40 -0400
Received: from smtp5.pp.htv.fi (smtp5.pp.htv.fi [213.243.153.39])	by
mx2.redhat.com (8.13.1/8.13.1) with ESMTP id l33BjcSD019557	for
<fedora-packaging redhat com>; Tue, 3 Apr 2007 07:45:39 -0400
Received: from viper.localdomain (cs181043142.pp.htv.fi
[82.181.43.142])	by smtp5.pp.htv.fi (Postfix) with ESMTP id
B4D425BC1E9	for <fedora-packaging redhat com>;	Tue,  3 Apr 2007 14:45:37
+0300 (EEST)
From: Ville Skyttä <ville skytta iki fi>
To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging redhat com>
Subject: Re: [Fedora-packaging] Hardlinking *.pyc and *.pyo
Date: Tue, 3 Apr 2007 14:45:37 +0300
User-Agent: KMail/1.9.6
References: <200704031342 25238 ville skytta iki fi>
<461237BB 5050700 leemhuis info>
In-Reply-To: <461237BB 5050700 leemhuis info>
MIME-Version: 1.0
Content-Type: text/plain;  charset="iso-8859-1"
Content-Disposition: inline
Message-Id: <200704031445 37318 ville skytta iki fi>
X-RedHat-Spam-Score: 0
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by
listman.util.phx.redhat.com id l33Bjfnb029939
X-loop: fedora-packaging redhat com
X-BeenThere: fedora-packaging redhat com
X-Mailman-Version: 2.1.5
Precedence: junk
Reply-To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging redhat com>
List-Id: Discussion of RPM packaging standards and practices for
Fedora Extras	<fedora-packaging.redhat.com>
List-Unsubscribe:
<https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request redhat com?subject=unsubscribe>
List-Archive: <https://www.redhat.com/archives/fedora-packaging>
List-Post: <mailto:fedora-packaging redhat com>
List-Help: <mailto:fedora-packaging-request redhat com?subject=help>
List-Subscribe:
<https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request redhat com?subject=subscribe>
Sender: fedora-packaging-bounces redhat com
Errors-To: fedora-packaging-bounces redhat com
X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on
basicbox7.server-home.net
X-Spam-Level:
X-Spam-Status: No, score=-0.3 required=5.0 tests=AWL
autolearn=disabled 	version=3.0.3
X-UIDL: GSK"!`f=!!:Wi"!p1+"!

On Tuesday 03 April 2007, Thorsten Leemhuis wrote:
> On 03.04.2007 12:42, Ville Skyttä wrote:
> > Worth it?  Other comments?
>
> Hehe, sounds interesting. Did you tell the Live-CD guys about it? I'd
> say they will probably very interested in something like this (they
> could run hardlink directly for now until we decide what we want to do).

Nope, just brought it up in last week's packaging meeting and now posted
here.
Feel free to forward.

Another thing they could be interested in (again on my FC6 x86_64):
$ /usr/sbin/hardlink -ncv /usr/share/doc 2>&1 | tail -n 1
Would save 6692864 (of which COPYING's share is about 3.2M)

-------- Original-Nachricht --------
Return-Path: <Thorsten_Leemhuis gmx de>
X-Original-To: rpmbuild localhost
Delivered-To: rpmbuild localhost thl home
Received: by truhe.thl.home (Postfix, from userid 500)	id EF7088117;
Tue,  3 Apr 2007 14:05:05 +0200 (CEST)
X-Original-To: thl localhost
Delivered-To: thl localhost thl home
Received: from truhe.thl.home (localhost [127.0.0.1])	by truhe.thl.home
(Postfix) with ESMTP id D99CE80DE	for <thl localhost>; Tue,  3 Apr 2007
14:05:05 +0200 (CEST)
X-Original-To: fedora leemhuis info
Delivered-To: web550p1 basicbox7 server-home net
Received: from leemhuis.info [195.137.212.29]	by truhe.thl.home with
POP3 (fetchmail-6.3.6)	for <thl localhost> (single-drop); Tue, 03 Apr
2007 14:05:05 +0200 (CEST)
Received: from hormel.redhat.com (hormel.redhat.com [209.132.177.30])	by
basicbox7.server-home.net (Postfix) with ESMTP id BE818374D4B	for
<fedora leemhuis info>; Tue,  3 Apr 2007 14:01:32 +0200 (CEST)
Received: from listman.util.phx.redhat.com
(listman.util.phx.redhat.com [10.8.4.110])	by hormel.redhat.com
(Postfix) with ESMTP	id 93CE072FB4; Tue,  3 Apr 2007 08:01:33 -0400 (EDT)
Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com
[172.16.52.254])	by listman.util.phx.redhat.com (8.13.1/8.13.1) with
ESMTP id	l33C1VoW032671 for
<fedora-packaging listman util phx redhat com>;	Tue, 3 Apr 2007 08:01:32
-0400
Received: from mx1.redhat.com (mx1.redhat.com [172.16.48.31])	by
int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l33C1UEY004042	for
<fedora-packaging redhat com>; Tue, 3 Apr 2007 08:01:30 -0400
Received: from junior.physik.fu-berlin.de
(IDENT:U2FsdGVkX184gQ/KwtoT7+xBQgH3zjb63w3ZGzlWGD4 junior physik fu-berlin de
[130.133.35.30])	by mx1.redhat.com (8.13.1/8.13.1) with ESMTP id
l33C1RN5026158	for <fedora-packaging redhat com>; Tue, 3 Apr 2007
08:01:27 -0400
Received: from neu-vpn.physik.fu-berlin.de ([160.45.32.209]
helo=neu.nirvana)	by mail2.atrpms.net with esmtps (TLSv1:AES256-SHA:256)
(Exim 4.66)	id 1HYhhE-0001uV-06; Tue, 03 Apr 2007 14:01:25 +0200
Received: from thimm by neu.nirvana with local (Exim 4.66)
(envelope-from <Axel Thimm ATrpms net>)	id 1HYhhC-0001x3-TQ; Tue, 03 Apr
2007 14:01:06 +0200
Date: Tue, 3 Apr 2007 14:01:06 +0200
From: Axel Thimm <Axel Thimm ATrpms net>
To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging redhat com>
Message-ID: <20070403120106 GD29856 neu nirvana>
References: <200704031342 25238 ville skytta iki fi>
MIME-Version: 1.0
In-Reply-To: <200704031342 25238 ville skytta iki fi>
User-Agent: Mutt/1.5.13 (2006-08-11)
X-Scanned: No viruses found.
X-Scan-Signature: fe522d7d72b3ce68d16a3279c0f8f490
X-RedHat-Spam-Score: 0
X-loop: fedora-packaging redhat com
Subject: [Fedora-packaging] Re: Hardlinking *.pyc and *.pyo
X-BeenThere: fedora-packaging redhat com
X-Mailman-Version: 2.1.5
Precedence: junk
Reply-To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging redhat com>
List-Id: Discussion of RPM packaging standards and practices for
Fedora Extras	<fedora-packaging.redhat.com>
List-Unsubscribe:
<https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request redhat com?subject=unsubscribe>
List-Archive: <https://www.redhat.com/archives/fedora-packaging>
List-Post: <mailto:fedora-packaging redhat com>
List-Help: <mailto:fedora-packaging-request redhat com?subject=help>
List-Subscribe:
<https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request redhat com?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============1626666452=="
Sender: fedora-packaging-bounces redhat com
Errors-To: fedora-packaging-bounces redhat com
X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on
basicbox7.server-home.net
X-Spam-Level:
X-Spam-Status: No, score=-0.2 required=5.0 tests=AWL
autolearn=disabled 	version=3.0.3
X-UIDL: ff="!/9`"! MP!!34]!!

On Tue, Apr 03, 2007 at 01:42:24PM +0300, Ville Skyttä wrote:
> Related to recent space saving discussions, I came across PLD's
> rpm-build-macros package recently, and found that they hardlink
> identical *.pyc and *.pyo.  In a lot of cases, they're the same,
> and there's some potential for saving some MB "for free",
> on my FC6 x86_64 box:
>
> $ /usr/sbin/hardlink -ncv /usr/lib*/python2.4 2>&1 | tail -n 1
> Would save 11116544

I get more than twice as much on a typical FC6/x86_64 system:
27275264. That's 26 MB on 166MB total, e.g. saving 16%.

# du -sc /usr/lib*/python2.4| tail -n 1
170200  total

On another system I get 21MB of 144MB total, e.g about 15%.

> The PLD implementation looks like this:
>
> # Hardlink binary identical .pyc and .pyo files
> # (idea by glen <at> pld-linux <dot> org)
> %__spec_install_post_py_hardlink {\
> %{!?no_install_post_py_hardlink: __spec_install_post_py_hardlink() { \
> [ ! -d "$RPM_BUILD_ROOT" ] || find "$RPM_BUILD_ROOT" -name '*.pyc' |
while read a; do \
>         b="$(echo $a|sed -e 's/.pyc$/.pyo/')"; \
>         if cmp -s "$a" "$b"; then \
>                 ln -f "$a" "$b"; \
>         fi; \
> done \
> }; __spec_install_post_py_hardlink } }
>
> The use of "cmp" would require diffutils installed.  Or the above
> could be converted to use hardlink instead (which would have to be made
> sure to be around) or maybe sha1sum (in coreutils, pretty much always
> around in buildroots).
>
> I suppose something like the above could be easily added to
> redhat-rpm-config or rpm, eg. embedded in brp-python-bytecompile
> or run after it in %__os_install_post.

brp-python-bytecompile sounds like the best spot since the pyc/pyos
are created there. Maybe we should ship an improved
brp-python-bytecompile in redhat-rpm-config while lobbying rpm
upstream to adopt it?

> Worth it?  Other comments?

A 15% space gain (under python) w/o any drawbacks? Always worth it. :)


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]