fedora-infrastructure

#9153 Combine uncombined http logs
Closed: Fixed by smooge. Opened by wwoods.

Closed: Fixed
Reopen Issue

Describe what you would like us to do:

Basically, run:

for ymd in 2020/06/{01..30}; do [ -d /mnt/fedora_stats/combined-http/$ymd ] || echo /usr/local/bin/combineHttpLogs.sh $ymd; done

on log01.

The colo move left gaps in the combined_http logs, which has lead to problems with the DNF Countme data. The July data all seems fine, but we're still missing a bunch of June. Running the above should fill in the gaps.

When do you need this to be done by? (YYYY/MM/DD)

Metadata Update from @pingou:
- Issue assigned to pingou
- Issue tagged with: low-trouble, medium-gain

Done :)

Metadata Update from @pingou:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

I did re-run it w/o the "echo" so the actual command gets run ;-)

It actually took a little while to run and here is the output:

ls: cannot access '/var/log/hosts/people*/2020/06/09/http//fedora*access.log.xz': No such file or directory
ls: cannot access '/var/log/hosts/people*/2020/06/10/http//fedora*access.log.xz': No such file or directory
ls: cannot access '/var/log/hosts/people*/2020/06/11/http//fedora*access.log.xz': No such file or directory

Sorry I have been working on this in the background. Most of the logs should have been combined but a second part which extracts data needed to be run.

Metadata Update from @smooge:
- Issue status updated to: Open (was: Closed)

Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: groomed

I did re-run it w/o the "echo" so the actual command gets run ;-)

Hah, oh right, should have pointed out that "safety feature". Thank you for figuring out what I meant!

It actually took a little while to run and here is the output:
ls: cannot access '/var/log/hosts/people/2020/06/09/http//fedoraaccess.log.xz': No such file or directory
ls: cannot access '/var/log/hosts/people/2020/06/10/http//fedoraaccess.log.xz': No such file or directory
ls: cannot access '/var/log/hosts/people/2020/06/11/http//fedoraaccess.log.xz': No such file or directory

oh, interesting. So it looks like we're actually missing logs for a few days? But.. only some of them?

Specifically, for the mirrors, 2020/06/{09..11} look much smaller than usual:

[wwoods@log01 combined-http][PROD-IAD2]$ du -h 2020/06/{07..13}/mirrors.*.log
5.0G    2020/06/07/mirrors.fedoraproject.org-access.log
5.0G    2020/06/08/mirrors.fedoraproject.org-access.log
1.6G    2020/06/09/mirrors.fedoraproject.org-access.log
1.6G    2020/06/10/mirrors.fedoraproject.org-access.log
1.9G    2020/06/11/mirrors.fedoraproject.org-access.log
5.4G    2020/06/12/mirrors.fedoraproject.org-access.log
5.4G    2020/06/13/mirrors.fedoraproject.org-access.log

Checking their sizes against the uncompressed logs, it seems like the source logs are just missing - or maybe we legit had fewer hits those days?

xz_totalsize() { size=$(xz --robot -l $1 | awk '/^totals/ {print $5}'); echo -e "$size\\t$1"; }
for ymd in 2020/06/{07..13}; do
    du -b /mnt/fedora_stats/combined-http/$ymd/mirrors.*access.log
    xz_totalsize "/var/log/hosts/proxy*/$ymd/http/mirrors.*access.log.xz"
    echo
done
5344948181      /mnt/fedora_stats/combined-http/2020/06/07/mirrors.fedoraproject.org-access.log
87031   /var/log/hosts/proxy*/2020/06/07/http/mirrors.*access.log.xz
5304609384      /mnt/fedora_stats/combined-http/2020/06/08/mirrors.fedoraproject.org-access.log
42042   /var/log/hosts/proxy*/2020/06/08/http/mirrors.*access.log.xz
1679923802      /mnt/fedora_stats/combined-http/2020/06/09/mirrors.fedoraproject.org-access.log
1679923802      /var/log/hosts/proxy*/2020/06/09/http/mirrors.*access.log.xz
1676739979      /mnt/fedora_stats/combined-http/2020/06/10/mirrors.fedoraproject.org-access.log
1676739979      /var/log/hosts/proxy*/2020/06/10/http/mirrors.*access.log.xz
1926123912      /mnt/fedora_stats/combined-http/2020/06/11/mirrors.fedoraproject.org-access.log
1926123912      /var/log/hosts/proxy*/2020/06/11/http/mirrors.*access.log.xz
5687690512      /mnt/fedora_stats/combined-http/2020/06/12/mirrors.fedoraproject.org-access.log
5687690512      /var/log/hosts/proxy*/2020/06/12/http/mirrors.*access.log.xz
5694784311      /mnt/fedora_stats/combined-http/2020/06/13/mirrors.fedoraproject.org-access.log
5694784311      /var/log/hosts/proxy*/2020/06/13/http/mirrors.*access.log.xz

So - are we missing logs in /var/log/hosts, or is that (as far as we know) the complete data for June 9-11?

I believe we had a lot less successful users during those days. We had a couple of outages right before we dropped the servers in PHX2 which was causing mirrormanager to not respond well. I have combined all the logs for June and July and I think the days with low counts are due to other outages we were fighting.

Metadata Update from @smooge:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Yeah, those were the actual heavy days of datacenter migration... so it could be stuff was working, but logging to the old log server and not the new one, etc.

Definitely worth explaining as it likely cases a blip in the stats, but I think we can chalk it up to the move.

Metadata

low-trouble medium-gain

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.