1# -*- coding: utf-8 -*- 2# Copyright 2013 Google Inc. All Rights Reserved. 3# 4# Licensed under the Apache License, Version 2.0 (the "License"); 5# you may not use this file except in compliance with the License. 6# You may obtain a copy of the License at 7# 8# http://www.apache.org/licenses/LICENSE-2.0 9# 10# Unless required by applicable law or agreed to in writing, software 11# distributed under the License is distributed on an "AS IS" BASIS, 12# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13# See the License for the specific language governing permissions and 14# limitations under the License. 15"""Additional help about CRC32C and installing crcmod.""" 16 17from __future__ import absolute_import 18 19from gslib.help_provider import HelpProvider 20 21_DETAILED_HELP_TEXT = (""" 22<B>OVERVIEW</B> 23 Google Cloud Storage provides a cyclic redundancy check (CRC) header that 24 allows clients to verify the integrity of object contents. For non-composite 25 objects GCS also provides an MD5 header to allow clients to verify object 26 integrity, but for composite objects only the CRC is available. gsutil 27 automatically performs integrity checks on all uploads and downloads. 28 Additionally, you can use the "gsutil hash" command to calculate a CRC for 29 any local file. 30 31 The CRC variant used by Google Cloud Storage is called CRC32C (Castagnoli), 32 which is not available in the standard Python distribution. The implementation 33 of CRC32C used by gsutil is provided by a third-party Python module called 34 `crcmod <https://pypi.python.org/pypi/crcmod>`_. 35 36 The crcmod module contains a pure-Python implementation of CRC32C, but using 37 it results in very poor performance. A Python C extension is also provided by 38 crcmod, which requires compiling into a binary module for use. gsutil ships 39 with a precompiled crcmod C extension for Mac OS X; for other platforms, see 40 the installation instructions below. 41 42 At the end of each copy operation, the gsutil cp and rsync commands validate 43 that the checksum of the source file/object matches the checksum of the 44 destination file/object. If the checksums do not match, gsutil will delete 45 the invalid copy and print a warning message. This very rarely happens, but 46 if it does, please contact gs-team@google.com. 47 48 49<B>CONFIGURATION</B> 50 To determine if the compiled version of crcmod is available in your Python 51 environment, you can inspect the output of the gsutil version command for the 52 "compiled crcmod" entry:: 53 54 $ gsutil version -l 55 ... 56 compiled crcmod: True 57 ... 58 59 If your crcmod library is compiled to a native binary, this value will be 60 True. If using the pure-Python version, the value will be False. 61 62 To control gsutil's behavior in response to crcmod's status, you can set the 63 "check_hashes" configuration variable. For details on this variable, see the 64 surrounding comments in your gsutil configuration file. If check_hashes is not 65 present in your configuration file, rerun gsutil config to regenerate the 66 file. 67 68 69<B>INSTALLATION</B> 70 CentOS, RHEL, and Fedora 71 ------------------------ 72 73 To compile and install crcmod: 74 75 sudo yum install gcc python-devel python-setuptools 76 sudo easy_install -U pip 77 sudo pip uninstall crcmod 78 sudo pip install -U crcmod 79 80 Debian and Ubuntu 81 ----------------- 82 83 To compile and install crcmod: 84 85 sudo apt-get install gcc python-dev python-setuptools 86 sudo easy_install -U pip 87 sudo pip uninstall crcmod 88 sudo pip install -U crcmod 89 90 Mac OS X 91 -------- 92 93 gsutil distributes a pre-compiled version of crcmod for OS X, so you shouldn't 94 need to compile and install it yourself. If for some reason the pre-compiled 95 version is not being detected, please let the Google Cloud Storage team know 96 (see "gsutil help support"). 97 98 To compile manually on OS X, you will first need to install 99 `XCode <https://developer.apple.com/xcode/>`_ and then run: 100 101 sudo easy_install -U pip 102 sudo pip install -U crcmod 103 104 Windows 105 ------- 106 107 An installer is available for the compiled version of crcmod from the Python 108 Package Index (PyPi) at the following URL: 109 110 https://pypi.python.org/pypi/crcmod/1.7 111 112 MSI installers are available for the 32-bit versions of Python 2.6 and 2.7. 113 Make sure to install to a 32-bit Python directory. If you're using 64-bit 114 Python it won't work with 32-bit crcmod, and instead you'll need to install 115 32-bit Python in order to use crcmod. 116 117 Note: If you have installed crcmod and gsutil hasn't detected it, it may have 118 been installed to the wrong directory. It should be located at 119 <python_dir>\\files\\Lib\\site-packages\\crcmod\\ 120 121 In some cases the installer will incorrectly install to 122 <python_dir>\\Lib\\site-packages\\crcmod\\ 123 124 Manually copying the crcmod directory to the correct location should resolve 125 the issue. 126""") 127 128 129class CommandOptions(HelpProvider): 130 """Additional help about CRC32C and installing crcmod.""" 131 132 # Help specification. See help_provider.py for documentation. 133 help_spec = HelpProvider.HelpSpec( 134 help_name='crc32c', 135 help_name_aliases=['crc32', 'crc', 'crcmod'], 136 help_type='additional_help', 137 help_one_line_summary='CRC32C and Installing crcmod', 138 help_text=_DETAILED_HELP_TEXT, 139 subcommand_help_text={}, 140 ) 141