Not Enough Disk Space on the Server Hosting the Update Manager - AFS

AFS Troubleshooting Guide

Product
AFS
AFS_Version
7.9
Platform
Debian
Category
Reference Guide

Issue

Content and search databases are piling up in the Update Manager, and some cleanup is needed.

Impact(s) on users

This issue has no impact on end-users.

Possible root cause(s)

This issue might occur when:

  • Services are pushing content in an incremental way, and all increments are kept in the Update Manager.
  • No cleanup is regularly performed, ant the content just accumulates in the Update Manager databases.

To detect the databases, whose size exceeds 1 Gigabyte, which need to be cleaned, run the following command:

du -hs /usr/local/afs7/bo-ac/update_manager/data/$AFS_VERSION/* | grep G

To determine which component(s) (agents => SEARCH, content => CREP_REPLY) and feeds are the largest ones, check the following elements:

  • The size of the different statuses
  • The agents and content directories
  • The component feeds

Then check the size on the Indexing server databases:

  • The size of result for SEARCH component
  • The size of the content repository MongoDB database for CREP_REPLY component.

To clean agents, clean up unnecessary Search databases from the Update Manager for a given service.

To clean content, clean up unnecessary Content databases from the Update Manager for a given service.

Solution(s)

Clean up unnecessary Search databases from the Update Manager for a given service

Pre-requisites

  • Check that all databases have enough disk space (the size of search database stored on the indexing server):

    df -h

Steps

  1. Put the reply synchronization on hold.
    1. List all Front servers configured to get content from this Update Manager server: Log in to the Back Office, go to Datacenter and check Search. The list of machines hosting a Query Manager will be displayed.
    2. Stop afs-host on all servers:

      As root user

      systemctl stop afs-host.service

  2. Pause the PaF of the concerned service:

    As antidot user

    /usr/local/afs7/bin/afs_pafmanager -P -s $SERVICE_ID -S $SERVICE_STATUS -f $SERVICE_FEED

  3. On the Update Manager server, delete unnecessary content for the concerned feed(s) using the following command:

    As antidot user

    /usr/local/afs7/bin/afs_um_delete -t SEARCH -s $SERVICE_ID -S $SERVICE_STATUS -f $SERVICE_FEED

  4. Put the PaFs of all the services on hold.
    1. List all the Indexing servers pushing to this Update Manager server: Log into the Back Office, go to Datacenter and check PaF Manager. The list of machines hosting a PaF Manager is displayed.
    2. Pause the PaFs of all the services:

      As antidot user

      /usr/local/afs7/bin/afs_pafmanager -P

  5. Restart the Update Manager.

    Note: Wait for the Update Manager transfers to complete. One way to check this is by us afs_um_top.

    As root user

    systemctl restart afs-updatemanager.service

  6. Deploy the search database from the Indexing server:

    As antidot user

    /usr/local/afs7/bin/afs_service_deploy -s $SERVICE_ID -S $SERVICE_STATUS -t SEARCH -f $SERVICE_FEED result

  7. Resume PaFs on all Indexing servers:

    /usr/local/afs7/bin/afs_pafmanager -R

  8. Resume the PaF on the concerned service:

    /usr/local/afs7/bin/afs_pafmanager -R -s $SERVICE_ID -S $SERVICE_STATUS -f $SERVICE_FEED

  9. On all Reply servers, restart afs-host:

    As root user

    systemctl start afs-host.service

Post-requisites

  • Check that all Reply servers are properly synchronized.

    This may take a while. While synchronizing there will be a temporary directory named working-<id> in the directory containing the reply database (db).

Clean up unnecessary Content databases from the Update Manager for a given service

The following procedure is not applicable to Nerada.

Pre-requisites

  • Check that all databases have enough disk space left:

    On the Indexing server, connect to Mongo and examine the size of the database you are interested in:

    mongo
    show dbs

    The Indexing server and the Update Manager server should have an equivalent free disk space.

    The Reply server should have a free disk space of at least twice the size of the MongoDB database.

    df -h

Steps

  1. Put the reply synchronization on hold.
    1. List all the Front servers configured to get content from the Update Manager server: Log into the Back Office, go to Datacenter and check Search, the list of machines hosting a Query Manager will be displayed.
    2. Stop afs-host on all the servers:

      As root user

      systemctl stop afs-host.service

  2. Pause the PaF of the concerned service:

    As antidot user

    /usr/local/afs7/bin/afs_pafmanager -P -s $SERVICE_ID -S $SERVICE_STATUS -f $SERVICE_FEED

  3. Change the AFS7 variable to the path to the PaF directory whose content must be exported.

    export AFS7=$PATH_TO_THE_PAF

  4. Start exporting the Content Repository (it may take a long time depending on the size of content).

    Run the following command from the $AFS7 directory.

    As antidot user

    /usr/local/afs7/bin/afs_crexport -v

    A new export directory is created. It is the directory to use in Step 8 of this procedure.

  5. On the Update Manager server, delete unnecessary content for all existing feed(s) with the following command:

    As antidot user

    /usr/local/afs7/bin/afs_um_delete -t CREP_REPLY -s $SERVICE_ID -S $SERVICE_STATUS -f $SERVICE_FEED

  6. Pause the PaFs of all the services:

    Note: List all the Indexing servers pushing to this Update Manager server: Log into the Back Office, go to Datacenter and check PaF Manager, the list of machines hosting a PaF Manager will be displayed.

    As antidot user

    /usr/local/afs7/bin/afs_pafmanager -P

  7. Restart the Update Manager.

    Note: Wait for the Update Manager transfers to complete. One way to check this is by using afs_um_top.

    As root user

    systemctl restart afs-updatemanager.service

  8. Deploy the content database from the Indexing server:

    As antidot user

    /usr/local/afs7/bin/afs_service_deploy -s $SERVICE_ID -S $SERVICE_STATUS -t CREP_REPLY -f $SERVICE_FEED $CREXPORT_DIRECTORY

    Where $CREXPORT_DIRECTORY is the directory generated in step 2.

  9. Resume PaFs on all Indexing servers:

    As antidot user

    /usr/local/afs7/bin/afs_pafmanager -R

  10. Resume the PaF on the concerned service:

    /usr/local/afs7/bin/afs_pafmanager -R -s $SERVICE_ID -S $SERVICE_STATUS -f $SERVICE_FEED

  11. On all Reply servers, restart afs_host:

    As root user

    systemctl start afs-host.service

Post-requisites

  • Check that all Reply servers are properly synchronized.

    This may take a while. While synchronizing, there will be a temporary directory named working-<id> in /usr/local/afs7/reply/$SERVICE_ID/$SERVICE_STATUS/content/$AGENT.

    When the synchronization is over, the data is written in Mongo.