1Release 4.15 (release date: 2015-09-08) 2======================================= 3Bug Fixes 4--------- 5- Fixed an OverflowError in apitools that caused download 6 failures for large files on 32-bit machines. 7- Removed unnecessary sending of range headers for downloads when 8 using the XML API. 9- Fixed a bug that caused perfdiag to report extremely high throughput 10 when the -p flag was unspecified and exactly one of the -c or -k flags 11 were specified. 12- Fixed a ValueError that occurred on Python 2.6 with sliced object downloads. 13 14Other Changes 15------------- 16- HTTP connections for downloads and uploads in the JSON API are now 17 re-used per-thread. 18- When gsutil's automatic update feature prompts and the user 19 chooses to update, gsutil will now exit with status code 1 after 20 updating (because the original command was not executed). 21- The cp -A flag is disabled when using gsutil -m to ensure that 22 ordering is preserved when copying between versioned buckets. 23 24Release 4.14 (release date: 2015-08-24) 25======================================= 26New Features 27------------ 28- Implemented Sliced Object Download feature. 29 This breaks up a single large object into multiple pieces and 30 downloads them in parallel, improving performance. The gsutil cp, mv 31 and rsync commands now use this by default when compiled crcmod 32 is available for performing fast end-to-end integrity checks. 33 If compiled crcmod is not available, normal object download will 34 be used. Sliced download can be used in conjunction with the global -m 35 flag for maximum performance to download multiple objects in 36 parallel while additionally slicing each object. 37 See the "SLICED OBJECT DOWNLOAD" section of "gsutil help cp" for 38 details. 39 Note: sliced download may cause performance degradation for disks 40 with very slow seek times. You can disable this feature by setting 41 sliced_object_download_threshold = 0 in your .boto configuration file. 42- Added rthru_file and wthru_file test modes to perfdiag, allowing 43 measurement of reads and writes from a disk. This also allows 44 measurement of transferring objects too large to fit in memory. 45 The size restriction of 20GiB has been lifted. 46- perfdiag now supports a -p flag to choose a parallelism strategy 47 (slice, fan, or both) when using multiple threads and/or processes. 48 49Bug Fixes 50--------- 51- Fixed an IOError that could occur in apitools when acquiring credentials 52 using multiple threads and/or processes on Google Compute Engine. 53- Fixed a bug where rm -r would attempt to delete a nonexistent bucket. 54- Fixed a bug where a default object ACL could not be set or changed to empty. 55- Fixed a bug where cached credentials corresponding to an old account could 56 be used (for example, credentials associated with a prior .boto 57 configuration file). 58- Fixed a bug in apitools for retrieving byte ranges of size 1 (for example, 59 "cat -r 1-1 ...") 60- Fixed a bug that caused the main gsutil process to perform all work leaving 61 all gsutil child processes idle. 62- Fixed a bug that caused multiple threads not to be used when 63 multiprocessing was unavailable. 64- Fixed a bug that caused rsync to skip files that start with "." when the 65 -r option was not used. 66- Fixed a bug that caused rsync -C to bail out when it failed to read 67 a source file. 68- Fixed a bug where gsutil stat printed unwanted output to stderr. 69- Fixed a bug where a parallel composite upload could return a nonzero exit 70 code even though the upload completed successfully. This occurred if 71 temporary component deletion triggered a retry but the original request 72 succeeded. 73- Fixed a bug where gsutil would exit with code 0 when both running in 74 debug mode and encountering an unhandled exception. 75- Fixed a bug where gsutil would suggest using parallel composite uploads 76 multiple times. 77 78Other Changes 79------------- 80- Bucket removal is now supported even if billing is disabled for that bucket. 81- Refactored Windows installs to no longer use any multiprocessing module 82 functions, as gsutil has never supported multiple processes on Windows. 83 Multithreading is unaffected and still available on Windows. 84- All downloads are now written to a temporary file with a "_.gstmp" suffix 85 while the download is still in progress. 86- Re-hashing of existing bytes when resuming downloads now displays progress. 87- Reduced the total number of multiprocessing.Manager processes to two. 88- The rm command now correctly counts the number of objects that could 89 not be removed. 90- Increased the default retries to match the Google Cloud Storage SLA. 91 By default, gsutil will now retry 23 times with exponential backoff up 92 to 32 seconds, for a total timespan of ~10 minutes. 93- Improved bucket subdirectory checks to a single HTTP call. Detection of 94 _$folder$ placeholder objects is now eventually consistent. 95- Eliminated two unnecessary HTTP calls when performing uploads via 96 the cp, mv, or rsync commands. 97- Updated documentation for several topics including acl, cache-control, 98 crcmod, cp, mb, rsync, and subdirs. 99- Added a warning about using parallel composite upload with NEARLINE 100 storage-class buckets. 101 102Release 4.13 (release date: 2015-06-03) 103======================================= 104New Features 105------------ 106- Added -U flag to cp and rsync commands to allow skipping of unsupported 107 object types. 108- Added support for Google Developer Shell credentials. 109 110Bug Fixes 111--------- 112- Precondition headers (x-goog-if-...) are now respected for the setmeta 113 command. 114- Fixed an index out of range error that could occur with an empty 115 parallel composite upload tracker file. 116- The stat command outputs errors to stderr instead of stdout. 117- Fixed two possible sources of ResumableUploadStartOverException from 118 httplib2 and oauth2client. 119- Fixed a bug in the compose command where a missing source object resulted 120 in an error message claiming the destination object was missing. 121 122Other Changes 123------------- 124- Added a help section on throttling gsutil. 125- Resumable uploads will now start over if a PUT to the upload ID returns 126 an HTTP 404. Previously this behavior applied only to an HTTP 410. 127- XML API resumable uploads now retry on HTTP 429 errors, matching the 128 behavior of JSON API resumable uploads. 129- Improved response to process kill signals, reducing the likelihood of 130 leaving orphaned child processes and temporary files. 131- Bucket lifecycle configuration now works for S3. 132- Removed the deprecated setmeta -n option. 133 134 135Release 4.12 (release date: 2015-04-20) 136======================================= 137New Features 138------------ 139- Added support for JSON-format service account private key files. 140- Added support for the Rewrite API (JSON API only). This is used for 141 all copies within the Google Cloud and supports copying objects across 142 storage classes and/or locations. 143 144Bug Fixes 145--------- 146- Fixed a bug that could cause downloads to have a hash mismatch (and deletion 147 of the corrupted file) when resumed across process breaks via a tracker file. 148 149Other Changes 150------------- 151- Updated documentation and examples for several topics including 152 acl, cp, dev, signurl, stat, and wildcards. 153 154 155Release 4.11 (release date: 2015-03-10) 156======================================= 157New Features 158------------ 159- Added Nearline storage class support to the mb command. 160 161Bug Fixes 162--------- 163- Fixed a bug for streaming uploads that could occasionally cause a HTTP 410 164 from the service or a hash mismatch (and deletion of the corrutped file). 165- Fixed an OverflowError that occurred when uploading files > 4GiB on a 32-bit 166 operating system. 167 168Other Changes 169------------- 170- Added documentation around using the Content-MD5 header to extend integrity 171 checking to include checksums computed by a client-side content pipeline. 172 173 174Release 4.10 (release date: 2015-03-03) 175======================================= 176Bug Fixes 177--------- 178- Fixed a bug that could cause undetected data corruption (preserving incorrect 179 data) if a streaming upload encountered a service error on non-8KiB-aligned 180 boundary. 181- Fixed a bug that caused downloads to be truncated if the connection broke, 182 resulting in a hash mismatch (and deletion of the corrupted file) for that 183 download. 184- Fixed a format string arguments error that occurred if a download exhausted 185 all retries. 186 187Other Changes 188------------- 189- The lifecycle command now accepts JSON input in the form of 190 "{ "lifecycle": { "rule" ..." in addition to "{ "rule": ...". 191- Improved access token expiry logic for GCE credentials. 192 193 194Release 4.9 (release date: 2015-02-13) 195======================================= 196New Features 197------------ 198- When using the JSON API, the ch acl/defacl subcommand now supports 199 project groups via the -p flag. For details, see "gsutil help acl ch". 200 201Bug Fixes 202--------- 203- Fixed a bug that caused daisy-chain copies (including cross-provider 204 copies) for files large than 100MiB to fail. 205- Fixed a bug that caused streaming uploads than ran for longer than 206 an hour to fail with HTTP 400s. 207- Fixed a bug where perfdiag would not properly clean up its test files. 208- Fixed a bug where using ls with the XML API could mistakenly report bucket 209 configuration as present. 210 211Other Changes 212------------- 213- Updated documentation for metadata, retries, security, and subdirs. 214- Tracker files are no longer written for small downloads. 215 216 217Release 4.8 (release date: 2015-01-23) 218======================================= 219New Features 220------------ 221- gsutil now supports HTTP proxy configuration via the http_proxy, 222 https_proxy, or HTTPS_PROXY environment variables. This configuration 223 is used only if proxy configuration is not present in the .boto 224 configuration file. 225- gsutil rsync now supports regex-based source and destination URL 226 exclusion via the -x flag. 227- The rm command now supports arguments on stdin via the -I flag. 228 229Bug Fixes 230--------- 231- Fixed a bug where perfdiag would fail if netstat was not available. 232- Fixed a bug where temporary ca_certs files were not being cleaned up. 233- Fixed a bug in rsync to unnecessarily remove or write objects, in some 234 cases leaving the destination in a non-synchronized state. 235 caused rsync to unnecessarily remove or rewrite objects. 236- Fixed a bug where rsync temporary listing files were not being 237 cleaned up when the rsync process was killed. 238- Fixed a bug where rsync would remove destination URLs if listing the 239 source encountered a non-retryable failure (for example, if the source 240 did not exist). 241- Fixed a bug where mv would fail for some Unicode filenames. 242- Fixed a bug where mv would remove the source URL after skipping the 243 destination URL. 244- Fixed a bug that caused daisy chain uploads to hang if the download thread 245 raised an exception. 246- Fixed a bug where acl ch would return a zero exit code even if it failed. 247- Fixed a bug that sometimes caused the progress display to render multiple 248 times at the end of an upload or download. 249 250Other Changes 251------------- 252- Resumable uploads of files using the JSON API now send their data in a 253 single request, making separate HTTP calls only when resuming is necessary. 254- The test command now runs tests in parallel by default, and test 255 parallelism on Windows is now supported. 256- All non-streaming downloads are now resumable (and retryable) by default, 257 regardless of size. 258- Canned ACLs and canned default object ACLs are now supported in the JSON 259 API (previously they would fall back to using the XML API). 260- Google Compute Engine service account credential tokens are now cached, 261 avoiding unnecessary refreshes. 262- Improved detection of the Google Compute Engine metadata server, 263 particularly when using the -m flag for multiprocessing. 264- Added new help sections about filename encoding and security/privacy 265 considerations. 266- Download progress is now displayed for small XML API downloads. 267 268 269Release 4.7 (release date: 2014-11-17) 270======================================= 271New Features 272------------ 273- Tab completion now works on gs:// URLs (for Cloud SDK installs only). 274 To install via Cloud SDK, see https://cloud.google.com/sdk/#Quick_Start 275- Streaming uploads (with source URL "-") using the JSON API now buffer 276 in-memory up to 100MB, allowing large streams to be retried in the event 277 of network flakiness or service errors. 278- Resumable uploads that receive a fatal service error (typically a 410) 279 are now automatically retried from the beginning. 280 281Bug Fixes 282--------- 283- Fixed an apitools bug that impacted upload performance and caused 284 "Retrying from byte __ after exception" to print after every 100MiB. 285- Fixed _$folder$ placeholder object detection on versioned buckets. 286- Removed an unnecessary credential check on load which increased 287 startup time by over one second in some cases. 288- SignURL now properly retries when checking if the signed object is 289 readable. 290- Files with both Content-Encoding and Content-Type gzip are now properly 291 removed when hash validation fails (only one of the two should be set). 292- The x-goog-if-generation-match:0 header now works properly with the XML API. 293- Fixed a bug that caused "defacl ch" on a bucket with a private ACL to fail. 294- The rm command now properly supports precondition headers. 295- Fixed a bug that caused large streaming uploads to fail with the message 296 "Failure: length too large" when using the JSON API. 297- Fixed a bug that caused JSON lifecycle configurations with createdBefore 298 conditions to fail with a DecodeError. 299 300Other Changes 301------------- 302- Byte counts now display accurate abbreviations of binary sizing. For example, 303 messages previously labeled MB are now properly labeled MiB to indicate 304 2**20 bytes. Only the labeling changed - the actual sizes were always binary. 305- Improved Cloud SDK integration, including improved error messages 306 and instructions. 307- The num_retries .boto configuration value now applies to all requests 308 (previously it was ignored for JSON API requests). 309- rsync now works with non-existent destination subdirectories. 310- Raised the default resumable upload threshold from 2MB to 8MB to 311 improve performance. 312- Benign retry messages now print only when debug mode is enabled via the 313 top-level -d flag. 314- The top-level -q flag now suppresses suggestions to use the -m flag. 315- Command synopsis is now output when the wrong number of arguments are 316 provided. 317- Removed dependency on google-api-python-client module, added dependencies on 318 oauth2client and six modules. 319 320Release 4.6 (release date: 2014-09-08) 321======================================= 322 323Bug Fixes 324--------- 325- Fixed a TypeError bug that occurred in perfdiag write throughput tests. 326- Fixed an rsync bug that caused invalid symlinks to abort the transfer 327 even when -e option was specified. 328- Fixed a perfdiag assumption that ipaddrlist was populated. 329- Fixed an AttributeError when setting an invalid canned ACL with defacl set. 330- Fixed a bug where non-resumable uploads would include payload in debug output 331 when for running in debug mode (-D). 332 333Other Changes 334------------- 335- Added the proxy_rdns configuration variable for clients that 336 do DNS lookups via a proxy. 337- Added the state_dir configuration variable for choosing the location of 338 gsutil's internal state files, including resumable transfer tracker files. 339 resumable_tracker_dir configuration variable is now deprecated. 340- Added DNS, connection latency, and proxy use information to perfdiag 341 command. 342- perfdiag command will not perform DNS lookups if they are disabled in 343 boto config. 344- perfdiag command will now only attempt to delete uploaded objects when 345 running write tests. 346- Added code coverage support to test command. 347- rsync -d now succeeds on a 404 for a to-be-deleted object (for example, when 348 the object was already deleted by an external process). 349 350Release 4.5 (release date: 2014-08-14) 351======================================= 352 353Bug Fixes 354--------- 355- Fixed a bug that caused resumable uploads to restart if gsutil was 356 terminated with CTRL-C. 357- Fixed a bug in defacl ch command that caused a failure when updating 358 an existing default object ACL entry. 359- Fixed an invalid literal bug during rsync file listing. 360- Made several improvements to JSON upload stability, including fixing a bug 361 which occasionally caused resumable upload hashes not to catch up properly. 362- All JSON calls now have socket timeouts, eliminating hangs under 363 flaky network conditions. 364- Fixed a bug where acl ch -g AllAuthenticatedUsers would instead add 365 AllUsers. 366- Fixed a bug that caused object custom metadata not to be preserved when 367 copying in the cloud. 368- Fixed a bug where du -s did not properly elide subdirectories. 369 370Other Changes 371------------- 372- Parallel composite uploads are now disabled by default until crcmod is 373 available in major Linux distributions. To re-enable the setting from 374 prior versions, in the [GSUtil] section of your .boto config file, add: 375 parallel_composite_upload_threshold=150M 376- Non-wildcarded URLs for existing objects now use Get before trying List 377 (as in gsutil3), and thus are not subject to eventual listing consistency. 378- gsutil -D now redacts proxy configuration values in the output. 379 380Release 4.4 (release date: 2014-07-17) 381======================================= 382 383New Features 384------------ 385- Added the hash command, which can calculate hashes of local files. 386 gsutil already calculates hashes for integrity checking, but this allows 387 the user to separately calculate the MD5 and CRC32c hashes of a local file. 388 389Bug Fixes 390--------- 391- Many improvements to JSON API media transfers, including far 392 more robust retry logic. 393- Fixed "File changed during upload: EOF..." errors on XML resumable uploads. 394- Fixed rsync command to read and write index files in binary mode. 395- Fix potential TypeError in _CheckAndHandleCredentialException. 396- Fixed possible data corruption when using JSON API uploads for 397 small files with lines starting with "From:", which would cause 398 integrity checks to fail. 399- Fixed gsutil cp to skip directory placeholders when downloading, avoiding 400 "directory exists where the file needs to be created" errors. 401- Fixed daisy chain cp/rsync for files >= 100MB. 402- Fixed a bug in JSON proxy support where the proxy info was sometimes unused. 403- Fixed a bug where an acl get on a private default object ACL returned an 404 error instead of a blank ACL. 405- Fixed a JSON API issue with large HTTP responses on 32-bit systems. 406 407Other Changes 408------------- 409- Improved object listing performance when using the XML API. 410- Improved various error messages. 411- Improved progress display during media transfer. 412- Switched to truncated exponential backoff for retries. 413- Improved OS-specific ulimit checks. 414- Added some information such as OS and Cloud SDK wrapping to gsutil version, 415 and changed the output format to be more uniform. 416- Daisy chain cp/rsync now supports resumable uploads. 417- Improved proxy support for proxy username and passwords. 418- x-amz headers are now supported for cp, rsync, and setmeta. x-amz-meta 419 headers continue to be supported as well. 420 421Release 4.3 (release date: 2014-06-10) 422======================================= 423 424Bug Fixes 425--------- 426- Fix acl/defacl ch changing the role of an existing group. 427- Fix unicode and 404 errors when using manifests. 428- Fix parallelism configuration bug that limited gsutil rsync to two threads 429 and could lead to rsync hangs. "gsutil -m rsync" runs much faster, and rsync 430 uploads of large local files are now faster via parallel composite upload. 431 Parallel composite uploads of large files are also faster. 432- Fix rsync bug with parallel composite uploads. 433- Fix TypeError that could occur when running the cp command with no 434 credentials. 435 436Other Changes 437------------- 438- Progress indicators for -m cp/rsync commands are now more readable. 439- Added command being run to gsutil -d/-D output. 440- Lowered default parallelism for 'gsutil -m test' and added hang detection. 441 442Release 4.2 (release date: 2014-06-05) 443======================================= 444 445New Features 446------------ 447- Added parallel test execution support to test command, ex: "gsutil -m test" 448 449Bug Fixes 450--------- 451- Fix failure during retry of an XML download. 452- Moved to boto release 2.29.1 fixing boto authentication erroneously 453 reporting OAuth2 credentials as invalid. 454- Fix parallel composite uploads when using only a single process and thread. 455- Fix an invalid seek during daisy chain operation that affected file copy 456 from Google Cloud Storage -> S3 for files greater than 8KB in size. 457- Fix "gsutil acl ch" with AllUsers or AllAuthenticatedUsers groups. 458- Fix some copy errors writing new lines to the manifest file. 459- Fix "gsutil test" return code to properly be 0 on success. 460 461Other Changes 462------------- 463- "gsutil cp -z" now ignores whitespace in the extension list. 464 465Release 4.1 (release date: 2014-05-28) 466======================================= 467 468Bug Fixes 469--------- 470- Fixed a bug in parallel composite uploads where uploads with 471 existing components would fail. 472- Moved gcs-oauth2-boto-plugin to version 1.5, fixing a bug in the PyPi gsutil 473 distribution that would cause gsutil to unnecessarily attempt to query 474 the Google Compute Engine metadata service. 475 476Other Changes 477------------- 478- Parallel composite uploads no longer specify an if-not-match precondition 479 when uploading component parts. 480- Parallel composite uploads no longer calculate a CRC32c hash prior to 481 uploading component parts (these are still validated by an MD5 hash). 482- Removed apitools dependency on gflags. 483 484Release 4.0 (release date: 2014-05-27) 485======================================= 486 487Major New Gsutil Version - Backwards-Incompatible Changes 488------------------------------ 489- The Google Cloud Storage configuration data supported by the acl, cors, 490 and lifecycle commands now uses the JSON format instead of the older XML 491 format. gsutil 4.0 will fail and provide conversion instructions if an XML 492 configuration file is provided as an argument for a gs:// URL. 493- gsutil no longer accepts arbitrary headers via the global -h flag. 494 Documented headers for gsutil commands are still supported; for the 495 full list of supported headers, see "gsutil help command_opts". 496- The compose command will now default the destination object's 497 Content-Type to the Content-Type of the first source object if none 498 is provided via the -h global flag. 499- The long-deprecated -t and -q options have been removed from the cp command. 500- The perfdiag command no longer supports adding a host header. 501- Having OAuth2 User Account credentials and OAuth2 Service Account 502 credentials configured simultaneously will now fail with an error message 503 to avoid confusion. Also, a single invalid credential will fail with an 504 error message. See "gsutil help creds" for details. 505- Bucket relocate scripts have been removed. 506- Downloading object names ending with '/' is no longer supported to avoid 507 problems this caused for directores using the Google Cloud Console. 508- rm -r now implies rm -ra (removing all object versions recursively). 509- All commands using the global -m option or a force option (such as 510 rm -f or cp -c) will now return a non-zero exit code if there are any 511 failures during the operation. 512- MD5 and CRC32c values are now represented in base64 encoding instead 513 of hex encoding (this includes manifest files). 514 515New Features 516------------ 517- The Google Cloud Storage JSON API (v1) is now the default API used 518 by gsutil for all commands targeting gs:// URLs. The JSON API is more 519 bandwidth efficient than the older XML API when transferring metadata 520 and does not require separate calls to preserve object ACLs when copying. 521 The XML API will automatically be used when accessing s3:// URLs. 522- The Google Cloud Storage XML API can be used in lieu of the JSON API 523 by setting 'prefer_api = xml' in the GSUtil section of your boto config file. 524- Added the rsync command that can synchronize cloud and local directories. 525- Added the signurl command that can generate Google Cloud Storage signed URLs. 526- The perfdiag command now supports a listing latency test. 527- The rb command now supports a -f flag allowing it to continue when errors 528 are encountered. 529- The test command now supports a -s flag that runs tests against S3. 530 531Other Changes 532------------- 533- All python files not under a third_party directory are now pylint-clean, 534 with the exception of TODO-format and a handful of warnings in root-level 535 files. As part of the de-linting process, many edge-case bugs were 536 identified and fixed. 537- The ls command now operates depth-first (as in Unix ls) instead 538 of breadth-first. 539- Daisy-chain copying does not currently support resumable uploads. 540- Several compatibility improvements for Windows and S3. 541 542 543Release 3.42 (release-date: 2014-01-15) 544======================================= 545 546Other Changes 547------------- 548 549- Fixed potential bug with update command on CentOS. 550 551 552Release 3.41 (release-date: 2014-01-14) 553======================================= 554 555Other Changes 556------------- 557 558- Changes to protect security of resumable upload IDs. 559 560 561Release 3.40: Skipped 562 563 564Release 3.39: Skipped 565 566 567Release 3.38 (release-date: 2013-11-25) 568 569Bug Fixes 570--------- 571 572- Fix to include version number in user-agent string. 573- Fix bug wherein -m flag or parallel uploads caused crash on systems without 574 /dev/shm. 575- Fix SSL errors and invalid results with perfdiag -c and -k rthru test. 576- Fixed cases where parallel composite uploads could leave orphaned components. 577- Fix bug attempting to stat objects you don't have auth to read. 578- Fixed bug breaking defacl's -d option. 579 580 581Other Changes 582------------- 583 584- Fixed gsutil config doc. 585- Fixed references to old command names; fix defacl ch example. 586- Improved error messages for deprecated command aliases. 587- Updated gsutil support info. 588 589 590New Features 591------------ 592 593- Enabled -R flag for recursion with setmeta command. 594 595 596Release 3.37 (release-date: 2013-09-25) 597======================================= 598 599Bug Fixes 600--------- 601 602- Fix parsing of -R for "acl ch" and chacl commands. 603- Fixed import statement of unittest2 which caused installations using Python 604 2.6 without unittest2 installed to fail when starting up gsutil. 605 606 607Other Changes 608------------- 609 610- Fixed tests so they pass on Windows and package installs. 611- Add a root logging handler manually instead of relying on basicConfig. 612- Fix apiclient import statement. 613- Exponential backoff for access token requests. 614- Fix flakiness in test TearDown to account for eventual consistency of object 615 listings. 616 617 618Release 3.36 (release-date: 2013-09-18) 619======================================= 620 621Bug Fixes 622--------- 623 624- Fix bug when a 400 or 403 exception has no detail. 625- Fix bugs with config -e and config -o. 626 627 628Other Changes 629------------- 630 631- Clarify stat command documentation regarding trailing slashes. 632- Add Generation and Metageneration to gsutil stat output. 633 634 635New Features 636------------ 637 638- Set config values from command line with -o. 639 640 641Release 3.35 (release-date: 2013-09-09) 642======================================= 643 644Bug Fixes 645--------- 646 647- Fix streaming upload to S3 and provide more useful stack traces multi-threaded failures. 648- Fix race condition in test_rm. 649- Fix retry decorator during test bucket cleanup. 650- Fixed cat bug that caused version to be ignored in URIs. 651- Don't decode -p or -h values other than x-goog-meta-. Fixes ability to use string project names. 652- Update bucket_relocate.sh to work on GCE. 653- Fix recursive uploading from subdirectories with unexpanded wildcard as source URI. 654- Make gsutil error text include <Message> content. 655- Change shebang line back to python because this doesn't work on some systems. 656- Fix hash_algs differences in perfdiag. 657- Update Python version check and shebang line. 658- Enforce project_id entry in config command; provide friendly error if missing proj ID. 659- Use transcoding-invariant headers when available in gs.Key. 660- Make gsutil cp not fail if unable to check versioning config on dest bucket. 661- Make gsutil detect when config fails because of proxy and prompt for proxy config. 662- Avoid checking metageneration attribute when long-listing S3 objects. 663- Exclude the no-op auth handler as indicating credentials are configured. 664 665 666New Features 667------------ 668 669- Implemented gsutil stat command. 670 671 672Other Changes 673------------- 674 675- Consolidate config-related commands. 676- Changed rm -r gs://bucket to delete bucket at end. 677- Various doc cleanup and improvement. 678- Warn user before updating to major new version. Also fixed minor version comparison bug, and added tests. 679- Change max component count to 1024. 680- Add retry-decorator as a submodule. 681- Explicitly state control chars to avoid in gsutil naming documentation. 682- Make config command recommend project strings. 683- Made long listing format a little better looking. 684- Allow --help flag for subcommands. 685- Implement help for subcommands and add OPTIONS sections for subcommands. 686- Add more detailed error message to notification watchbucket command. 687- Add notification URL configuration for notification tests. 688- Refactor to use upstream retry_decorator as external dependency. 689- Distribute cacerts file with gsutil. 690- Updated gsutil help to point to Google Cloud Console instead of older APIs console. 691- Make gsutil pass bundled cacerts.txt to oauth2client; stop checking SHA1 of certs, now that we no longer depend on boto distribution. 692- Move all TTY checks to a common util function and mock it for update tests. 693- Fix duplicate entry created in .gitmodules. 694- Fix unit test breakage because VERSION file is old. 695- Fix test using ? glob with ObjectToURI. 696- Fix update tests that fail for package installs. 697- Change bucket delete teardown to try more times. 698- Fix tests that perform operations on bucket listings. 699- Keep package install set to True unless VERSION file doesn't exist. 700- Fix handling of non-numeric version strings in update test. 701 702 703Release 3.34 (release-date: 2013-07-18) 704======================================= 705 706Bug Fixes 707--------- 708 709- Fixed a bug where the no-op authentication handler was being loaded after 710 other authentication plugins, causing the no-op handler to be chosen instead 711 of other valid credentials. 712 713 714Release 3.33 (release-date: 2013-07-16) 715======================================= 716 717Bug Fixes 718--------- 719 720- Added .git* to MANIFEST.in excludes and fixed cp doc typo. This was needed to 721 overcome problem caused by accidental inclusion of .git* files in release, 722 which caused the update command no longer to allow updates (since starting 723 in 3.32 it checks whether the user has any extraneous files in the gsutil 724 directory before updating) 725 726 727Release 3.32 (release-date: 2013-07-16) 728======================================= 729 730New Features 731------------ 732 733- Added support for getting and setting lifecycle configuration for buckets. 734- Implemented Parallel Composite Uploads. 735- Added a new du command that displays object size, similar to Linux du. 736 737 738Bug Fixes 739--------- 740 741- Fixed a bug when using ls -R on objects with trailing slashes. Closes #93. 742- Fixed so won't crash in perfdiag when nslookup is missing or gethostbyname 743 fails. 744- Smartly compare version strings during autoupdate check. 745- Made header handling for upload case-insensitive. 746- Re-enabled software update check for users with no credentials configured. 747- Fixed incorrectly-generated password editing comment in service account 748 config. Fixes #146. 749 750 751Other Changes 752------------- 753 754- Improved flow when encounter auth failure for GCE service account with no 755 configured storage scopes: 756 1. Changed HasConfiguredCredentials() logic not to include 757 has_auth_plugins as part of the evaluated expression, since that will 758 always evaluate to true under GCE (since GCE configures its internal 759 service account plugin under /etc/boto.cfg). 760 2. Changed ConfigureNoOpAuthIfNeeded logic so we configure no-op auth 761 plugin even if there is a config_file list, since GCE always configures 762 /etc/boto.cfg, even if user has no storage scopes configured. 763 3. Additional changes: 764 a. Removed assertion of oauth access token cache check log from 765 test_Doption.py, which may not be true sometimes (e.g., if user is 766 using HMAC creds). 767 b. Removed remnants of CONFIG_REQUIRED left over from earlier CL. 768 c. Merged dupe _ConfigureNoOpAuthIfNeeded functions from two code files, 769 moved to util.py. 770- Fixed confusing gsutil rm "Omitting" message. 771- Wrapped long gsutil update message. 772- Silenced additional possible perfdiag errors. 773- Improved perfdiag performance by only generating one chunk of random file. 774- Changed to swallow broken pipe errors when piping gsutil to other programs. 775- Made DotfulBucketNameNotUnderTld error message more user friendly. 776- Extracted function for building ACL error text from main try/except loop, 777 for better readability. 778- Disallowed gsutil update when user data present in gsutil dir. 779- Plumbed accept-encoding into HEAD requests in ls -L command. 780- Updated README and moved ReleaseNotes.txt to CHANGES.md. 781- Updated crcmod docs with link to Windows installer. 782- Updated documentation regarding gzip content-encoding. 783- Removed StorageUri parse check for lone ':' (interferes with using filenames 784 containing ':') 785- Added tests for gsutil update check and fixed bug for bad file contents. 786- Set accept-encoding and handle gzip on-the-fly encoding. 787 788 789Release 3.31 (release-date: 2013-06-10) 790======================================= 791 792New Features 793------------ 794 795- Implemented consumption of manifest files for cp. 796- Add ETag to ls -l and make ls -b variants more efficient. 797- Expand the manifest path to allow for tildes in paths. 798- Added bucket_relocate.sh script to gsutil. 799 800 801Bug Fixes 802--------- 803 804- Fix gsutil cp -R to copy all versioned objects. 805- Fixed bug where gsutil cp -D -n caused precondition failure. 806- Fixed gsutil daisy-chain copy to allow preserving ACLs when copying within 807 same provider. 808- Fix identification of non-MD5 ETags. 809- Fixed bugs where gsutil -q cp and gsutil cp -q sometimes weren't quiet. 810- Fixed unicode error when constructing tracker filename from non-ASCII 811 Unicode filename. 812- Fixed that noclobber would not resume partial resumable downloads. 813- Fixed bug when running gsutil cp -Dp by user other than object owner. 814- Properly encode metageneration and etag in ls output with -a and -e. 815- Update resumable threshold stated in gsutil help prod. 816 817 818Other Changes 819------------- 820 821- ls -Lb no longer shows total # files/total size of bucket, so that ls -Lb 822 instead provides an efficient way to view just the metadata for large buckets. 823- Catch and ignore EEXIST error when creating gsutil tracker dir. 824- Add note to gsutil update doc about auto-update checks being disabled with 825 gsutil -q option. 826- Disable hashing and increase buffer size in perfdiag. 827- Added better error messages for service account auth. 828- Make perfdiag behave more like normal gsutil, with multi-threading option. 829- Changed so auto-update check/prompt aren't made if gsutil -q specified. 830- Changed gsutil mb command to clarify that EU means European Union. 831- Added doc warnings about losing version ordering if using gsutil -m cp 832 between versioned buckets; removed trailing whitespace. 833- Added to gsutil cp -L doc to describe how to build a reliable script for 834 copying many objects. 835 836 837Release 3.30 (release-date: 2013-06-10) 838======================================= 839 840- Abandoned. 841 842 843Release 3.29 (release-date: 2013-05-13) 844======================================= 845 846Bug Fixes 847--------- 848 849- Fixed incorrect package installation detection that resulted in not being 850 able to run the update command while running gsutil from a symlink. 851 852Other Changes 853------------- 854 855- Added a test for debug mode (gsutil -D) output. 856- List numbering and title case fixes in additional help pages. 857- Removed dateutil module dependency from cp command test. 858- Updated documentation to clarify that public-read objects are cached for 1 859 hour by default. 860- Added a filter to suppress "module was already imported" warnings that were 861 sometimes printed while running gsutil on Google Compute Engine instances. 862 863 864Release 3.28 (release-date: 2013-05-07) 865======================================= 866 867New Features 868------------ 869 870- Added support for new Object Change Notifications feature. 871 872Bug Fixes 873--------- 874 875- Fixed problem where gsutil update command didn’t take default action. 876- Fixed a problem with the update command sometimes triggering an additional 877 update command. 878 879Other Changes 880------------- 881 882- Add packaging information to version output. 883- Removed fancy_urllib, since it is no longer used. 884- Changed num_retries default for resumable downloads to 6. 885- Don’t check for newer software version if gs_host is specified in boto 886 config file. 887- Modified oauth2client logging behavior to be consistent with gsutil. 888- Added gs_port configuration option. 889- Skip update tests when SSL is disabled. 890 891 892Release 3.27 (release-date: 2013-04-25) 893======================================= 894 895New Features 896------------ 897 898- Added a human readable option (-h) to ls command. 899- Changed WildcardIterator not to materialize list of all matching files from 900 directory listing (so works faster when walking over large directories) 901- Added -f option to setacl command to allow command to continue after errors 902 encountered. 903- Add manifest log support for the cp command. 904- Added never option for check_hashes_config; fixed bug that assumes an ETag 905 is always returned from server. 906- Made gsutil provide friendlier error message if attempting non-public data 907 access with missing credentials. 908- Set 70 second default socket timeout for httplib. 909- Add ability to run a single test class or function with the test command. 910 911Bug Fixes 912--------- 913 914- Don't check for updates if the user has no credentials configured. This 915 fixes a bug for users without credentials trying to use gsutil for first 916 time. 917- Fixed case where chacl command incorrectly recognized an email address as a 918 domain. 919- Fix setmeta command for S3 objects. 920- Fixed bug where wildcarded dest URI attempted string op on Key object. 921- Fixed case where gsutil -q outputted progress output when doing a streaming 922 upload. 923- Error handling for out of space during downloads. 924- Include ISO 8601-required "Z" at end of timestamp string for gsutil ls -l, 925 to be spec-compliant. 926- Removed deprecated setmeta syntax and fixed unicode issues. 927- Changed update command not to suggest running sudo if running under Cygwin. 928- Removed references to deprecated gs-discussion forum from gsutil built-in 929 help. 930- Add literal quotes around CORS config example URL in gsutil setcors help to 931 avoid having example URL turn into an HREF in auto-generated doc. 932 933Other Changes 934------------- 935 936- Added proper setup.py to make gsutil installable via PyPi. 937- Added warning to gsutil built-in help that delete operations cannot be 938 undone. 939- Replaced gsutil's OAuth2 client implementation with oauth2client. 940- Updates to perfdiag. 941- Updated config help about currently supported settings. 942- Fixes to setup.py and modified version command. 943- Move gslib/commands/cred_types.py to gslib, so only Command subclasses live 944 in gslib/commands. 945- Updated gsutil setmeta help no longer to warn that setmeta with versioning 946 enabled creates a new object. 947 948 949Release 3.26 (release-date: 2013-03-25) 950======================================= 951 952New Features 953------------ 954 955- Added support for object composition. 956- Added support for external service accounts. 957- Changed gsutil to check for available updates periodically (only while 958 stdin, stderr, stdout are connected to a TTY, so as not to interfere with 959 cron jobs). 960- Added chdefacl command. 961- Made gsutil built-in help available under 962 https://developers.google.com/storage/docs/gsutil 963- Add a command suggestion when the command name is not found. 964- Added byte suffix parsing to the -s parameter of perfdiag. 965- Added --help support to subcommands. Fixes #96. 966- Updated perfdiag command to track availability and record TCP settings. 967- Added metadata parameter to perfdiag command. 968- Added support for specifying byte range to cat command. 969- Output more bucket metadata on ls -Lb. 970- Implemented gsutil -q (global quiet) option (fixes issue #130). Also changed 971 gsutil to output all progress indicators using logging levels. Also changed 972 help command not to output bold escape sequences and not use PAGER if stdout 973 is not a tty, which also fixes bug that caused gsutil help test to fail. 974- Plumbed https_validate_certificates through to OAuth2 plugin handler, 975 allowing control over cert validation for OAuth2 requests 976- Fixed ISO 639.1 ref in config command help text 977 978Bug Fixes 979--------- 980 981- Fixed bug where gsutil cp -D didn't preserve metadata 982- Fixed problem where gsutil -m is hard to interrupt (partial fix for issue 983 #99 - only for Linux/MacOS; problem still exists for Windows). 984- Fixed broken reference to boto_lib_dir in update command. 985- Made changing ACL not retry on 400 error. 986- Fixed name expansion bug for case where uri_strs is itself an iterator 987 (issue #131); implemented additional naming unit test for this case. 988- Fixed flaky gsutil rm test 989- Fixed a bug in the chacl command that made it so you couldn't delete the 990 AllAuthenticatedUsers group from an ACL. 991 992Other Changes 993------------- 994 995- Refactored gsutil main function into gslib, with gsutil being a thin 996 wrapper. 997- Added a test for the update command. 998- Renamed gsutil meta_generation params to metageneration, for consistency 999 with GCS docs. 1000- Removed .pyc files from tarball/zipfile. 1001- Added new root certs to cacerts.txt, to provide additional flexibility 1002 in the future. 1003 1004 1005Release 3.25 (release-date: 2013-02-21) 1006======================================= 1007 1008Bug Fixes 1009--------- 1010 1011- Fixed two version-specific URI bugs: 1012 1. gsutil cp -r gs://bucket1 gs://bucket2 would create objects in bucket2 1013 with names corresponding to version-specific URIs in bucket1 (e.g., 1014 gs://bucket2/obj#1361417568482000, where the "#1361417568482000" part was 1015 part of the object name, not the object's generation). 1016 1017 This problem similarly caused gsutil cp -r gs://bucket1 ./dir to create 1018 files names corresponding to version-specific URIs in bucket1. 1019 2. gsutil rm -a gs://bucket/obj would attempt to delete the same object 1020 twice, getting a NoSuchKey error on the second attempt. 1021 1022 1023Release 3.24 (release-date: 2013-02-19) 1024======================================= 1025 1026Bug Fixes 1027--------- 1028 1029- Fixed bug that caused attempt to dupe-encode a unicode filename. 1030 1031Other Changes 1032------------- 1033 1034- Refactored retry logic from setmeta and chacl to use @Retry decorator. 1035- Moved @Retry decorator to third_party. 1036- Fixed flaky tests. 1037 1038 1039Release 3.23 (release-date: 2013-02-16) 1040======================================= 1041 1042Bug Fixes 1043--------- 1044 1045- Make version-specific URI parsing more robust. This fixes a bug where 1046 listing buckets in certain cases would result in the error 1047 'BucketStorageUri' object has no attribute 'version_specific_uri' 1048 1049 1050Release 3.22 (release-date: 2013-02-15) 1051======================================= 1052 1053New Features 1054------------ 1055 1056- Implemented new chacl command, which makes it easy to add and remove bucket 1057 and object ACL grants without having to edit XML (like the older setacl 1058 command). 1059- Implemented new "daisy-chain" copying mode, which allows cross-provider 1060 copies to run without buffering to local disk, and to use resumable uploads. 1061 This copying mode also allows copying between locations and between storage 1062 classes, using the new gsutil cp -D option. (Daisy-chain copying is the 1063 default when copying between providers, but must be explicitly requested for 1064 the other cases to keep costs and performance expectations clear.) 1065- Implemented new perfdiag command to run a diagnostic test against 1066 a bucket, collect system information, and report results. Useful 1067 when working with Google Cloud Storage team to resolve questions 1068 about performance. 1069- Added SIGQUIT (^\) handler, to allow breakpointing a running gsutil. 1070 1071Bug Fixes 1072--------- 1073 1074- Fixed bug where gsutil setwebcfg signature didn't match with 1075 HMAC authentication. 1076- Fixed ASCII codec decode error when constructing tracker filename 1077 from non-7bit ASCII input filename. 1078- Changed boto auth plugin framework to allow multiple plugins 1079 supporting requested capability, which fixes gsutil exception 1080 that used to happen where a GCE user had a service account 1081 configured and then ran gsutil config. 1082- Changed Command.Apply method to be resilient to name expansion 1083 exceptions. Before this change, if an exception was raised 1084 during iteration of NameExpansionResult, the parent process 1085 would immediately stop execution, causing the 1086 _EOF_NAME_EXPANSION_RESULT to never be sent to child processes. 1087 This resulted in the process hanging forever. 1088- Fixed various bugs for gsutil running on Windows: 1089 - Fixed various places from a hard-coded '/' to os.sep. 1090 - Fixed a bug in the cp command where it was using the destination 1091 URI's .delim property instead of the source URI. 1092 - Fixed a bug in the cp command's _SrcDstSame function by 1093 simplifying it to use os.path.normpath. 1094 - Fixed windows bug in tests/util.py _NormalizeURI function. 1095 - Fixed ZeroDivisionError sometimes happening during unit tests 1096 on Windows. 1097 1098- Fixed gsutil rm bug that caused exit status 1 when encountered 1099 non-existent URI. 1100- Fixed support for gsutil cp file -. 1101- Added preconditions and retry logic to setmeta command, to 1102 enforce concurrency control. 1103- Fixed bug in copying subdirs to subdirs. 1104- Fixed cases where boto debug_level caused too much or too little 1105 logging: 1106 - resumable and one-shot uploads weren't showing response headers 1107 when connection.debug > 0. 1108 - payload was showing up in debug output when connection.debug 1109 < 4 for streaming uploads. 1110 1111- Removed XML parsing from setacl. The previous implementation 1112 relied on loose XML handling, which could truncate what it sends 1113 to the service, allowing invalid XML to be specified by the 1114 user. Instead now the ACL XML is passed verbatim and we rely 1115 on server-side schema enforcement. 1116- Added user-agent header to resumable uploads. 1117- Fixed reporting bits/s when it was really bytes/s. 1118- Changed so we now pass headers with API version & project ID 1119 to create_bucket(). 1120- Made "gsutil rm -r gs://bucket/folder" remove xyz_$folder$ object 1121 (which is created by various GUI tools). 1122- Fixed bug where gsutil binary was shipped with protection 750 1123 instead of 755. 1124 1125Other Changes 1126------------- 1127 1128- Reworked versioned object handling: 1129 - Removed need for commands to specify -v option to parse 1130 versions. Versioned URIs are now uniformly handled by all 1131 commands. 1132 - Refactored StorageUri parsing that had been split across 1133 storage_uri and convenience; made versioned URIs render with 1134 version string so StorageUri is round-trippable (boto change). 1135 - Implemented gsutil cp -v option for printing the version-specific 1136 URI that was just created. 1137 - Added error detail for attempt to delete non-empty versioned 1138 bucket. Also added versioning state to ls -L -b gs://bucket 1139 output. 1140 - Changed URI parsing to use pre-compiled regex's. 1141 - Other bug fixes. 1142 1143- Rewrote/deepened/improved various parts of built-in help: 1144 - Updated 'gsutil help dev'. 1145 - Fixed help command handling when terminal does not have the 1146 number of rows set. 1147 - Rewrote versioning help. 1148 - Added gsutil help text for common 403 AccountProblem error. 1149 - Added text to 'gsutil help dev' about legal agreement needed 1150 with code submissions. 1151 - Fixed various other typos. 1152 - Updated doc for cp command regarding metadata not being 1153 preserved when copying between providers. 1154 - Fixed gsutil ls command documentation typo for the -L option. 1155 - Added HTTP scheme to doc/examples for gsutil setcors command. 1156 - Changed minimum version in documentation from 2.5 to 2.6 since 1157 gsutil no longer works in Python 2.5. 1158 - Cleaned up/clarify/deepen various other parts of gsutil 1159 built-in documentation. 1160 1161- Numerous improvements to testing infrastructure: 1162 - Completely refactored infrastructure, allowing deeper testing 1163 and more readable test code, and enabling better debugging 1164 output when tests fail. 1165 - Moved gslib/test_*.py unit tests to gslib/tests module. 1166 - Made all tests (unit and integration, per-command and modules 1167 (like naming) run from single gsutil test command. 1168 - Moved TempDir functions from GsUtilIntegrationTestCase to 1169 GsUtilTestCase. 1170 - Made test runner message show the test function being run. 1171 - Added file path support to ObjectToURI function. 1172 - Disabled the test command if running on Python 2.6 and unittest2 1173 is not available instead of breaking all of gsutil. 1174 - Changed to pass GCS V2 API and project_id from boto config 1175 if necessary in integration_testcase#CreateBucket(). 1176 - Fixed unit tests by using a GS-specific mocking class to 1177 override the S3 provider. 1178 - Added friendlier error message if test path munging fails. 1179 - Fixed bug where gsutil test only cleaned up first few test files. 1180 - Implemented setacl integration tests. 1181 - Implemented StorageUri parsing unit tests. 1182 - Implemented test for gsutil cp -D. 1183 - Implemented setacl integration tests. 1184 - Implemented tests for reading and seeking past end of file. 1185 - Implemented and tests for it in new tests module. 1186 - Changed cp tests that don't specify a Content-Type to check 1187 for new binary/octet-stream default instead of server-detected 1188 mime type. 1189 1190- Changed gsutil mv to allow moving local files/dirs to the cloud. 1191 Previously this was disallowed in the belief we should be 1192 conservative about deleting data from local disk but there are 1193 legitimate use cases for moving data from a local dir to the 1194 cloud, it's clear to the user this would remove data from the 1195 local disk, and allowing it makes the tool behavior more 1196 consistent with what users would expect. 1197- Changed gsutil update command to insist on is_secure and 1198 https_validate_certificates. 1199- Fixed release no longer to include extraneous boto dirs in 1200 top-level of gsutil distribution (like bin/ and docs/). 1201- Changed resumable upload threshold from 1 MB to 2 MB. 1202- Removed leftover cloudauth and cloudreader dirs. Sample code 1203 now lives at https://github.com/GoogleCloudPlatform. 1204- Updated copyright notice on code files. 1205 1206 1207Release 3.21 (release-date: 2012-12-10) 1208======================================= 1209 1210New Features 1211------------ 1212 1213- Added the ability for the cp command to continue even if there is an 1214 error. This can be activated with the -c flag. 1215- Added support for specifying src args for gsutil cp on stdin (-I option) 1216 1217Bug Fixes 1218--------- 1219 1220- Fixed gsutil test cp, which assumed it was run from gsutil install dir. 1221- Mods so we send generation subresource only when user requested 1222 version parsing (-v option for cp and cat commands). 1223 1224Other Changes 1225------------- 1226 1227- Updated docs about using setmeta with versioning enabled. 1228- Changed GCS endpoint in boto to storage.googleapis.com. 1229 1230 1231Release 3.20 (release-date: 2012-11-30) 1232======================================= 1233 1234New Features 1235------------ 1236 1237- Added a noclobber (-n) setting for the cp command. Existing objects/files 1238 will not be overwritten when using this setting. 1239 1240Bug Fixes 1241--------- 1242 1243- Fixed off-by-one error when reporting bytes transferred. 1244 1245Other Changes 1246------------- 1247 1248- Improved versioning support for the remove command. 1249- Improved test runner support. 1250 1251 1252Release 3.19 (release-date: 2012-11-26) 1253======================================= 1254 1255New Features 1256------------ 1257 1258- Added support for object versions. 1259- Added support for storage classes (including Durable Reduced Availability). 1260 1261Bug Fixes 1262--------- 1263- Fixed problem where cp -q prevented resumable uploads from being performed. 1264- Made setwebcfg and setcors tests robust wrt XML formatting variation. 1265 1266Other Changes 1267------------- 1268 1269- Incorporated vapier@ mods to make version command not fail if CHECKSUM file 1270 missing. 1271- Refactored gsutil such that most functionality exists in boto. 1272- Updated gsutil help dev instructions for how to check out source. 1273 1274 1275Release 3.18 (release-date: 2012-09-19) 1276======================================= 1277 1278Bug Fixes 1279--------- 1280 1281- Fixed resumable upload boundary condition when handling POST request 1282 when server already has complete file, which resulted in an infinite 1283 loop that consumed 100% of the CPU. 1284- Fixed one more place that outputted progress info when gsutil cp -q 1285 specified (during streaming uploads). 1286 1287Other Changes 1288------------- 1289 1290- Updated help text for "gsutil help setmeta" and "gsutil help metadata", to 1291 clarify and deepen parts of the documentation. 1292 1293 1294Release 3.17 (release-date: 2012-08-17) 1295======================================= 1296 1297Bug Fixes 1298--------- 1299 1300- Fixed race condition when multiple threads attempt to get an OAuth2 refresh 1301 token concurrently. 1302 1303Other Changes 1304------------- 1305 1306- Implemented simplified syntax for setmeta command. The old syntax still 1307 works but is now deprecated. 1308- Added help to gsutil cp -z option, to describe how to change where temp 1309 files are written. 1310 1311 1312Release 3.16 (release-date: 2012-08-13) 1313======================================= 1314 1315Bug Fixes 1316--------- 1317 1318- Added info to built-in help for setmeta command, to explain the syntax 1319 needed when running from Windows. 1320 1321 1322Release 3.15 (release-date: 2012-08-12) 1323======================================= 1324 1325New Features 1326------------ 1327 1328- Implemented gsutil setmeta command. 1329- Made gsutil understand bucket subdir conventions used by various tools 1330 (like GCS Manager and CloudBerry) so if you cp or mv to a subdir you 1331 created with one of those tools it will work as expected. 1332- Added support for Windows drive letter-prefaced paths when using Storage 1333 URIs. 1334 1335Bug Fixes 1336--------- 1337 1338- Fixed performance bug when downloading a large object with Content- 1339 Encoding:gzip, where decompression attempted to load the entire object 1340 in memory. Also added "Uncompressing" log output if file is larger than 1341 50M, to make it clear the download hasn't stalled. 1342- Fixed naming bug when performing gsutil mv from a bucket subdir to 1343 and existing bucket subdir. 1344- Fixed bug that caused cross-provider copies into Google Cloud Storage to 1345 fail. 1346- Made change needed to make resumable transfer progress messages not print 1347 when running gsutil cp -q. 1348- Fixed copy/paste error in config file documentation for 1349 https_validate_certificates option. 1350- Various typo fixes. 1351 1352Other Changes 1353------------- 1354 1355- Changed gsutil to unset http_proxy environment variable if it's set, 1356 because it confuses boto. (Proxies should instead be configured via the 1357 boto config file.) 1358 1359 1360Release 3.14 (release-date: 2012-07-28) 1361======================================= 1362 1363New Features 1364------------ 1365 1366- Added cp -q option, to support quiet operation from cron jobs. 1367- Made config command restore backed up file if there was a failure or user 1368 hits ^C. 1369 1370Bug Fixes 1371--------- 1372 1373- Fixed bug where gsutil cp -R from a source directory didn't generate 1374 correct destination path. 1375- Fixed file handle leak in gsutil cp -z 1376- Fixed bug that caused cp -a option not to work when copying in the cloud. 1377- Fixed bug that caused '/-' to be appended to object name for streaming 1378 uploads. 1379- Revert incorrect line I changed in previous CL, that attempted to 1380 get fp from src_key object. The real fix that's needed is described in 1381 https://github.com/GoogleCloudPlatform/gsutil/issues/72. 1382 1383Other Changes 1384------------- 1385 1386- Changed logging to print "Copying..." and Content-Type on same line; 1387 refactored content type and log handling. 1388 1389 1390Release 3.13 (release-date: 2012-07-19) 1391======================================= 1392 1393Bug Fixes 1394--------- 1395 1396- Included the fix to make 'gsutil config' honor BOTO_CONFIG environment 1397 variable (which was intended to be included in Release 3.12) 1398 1399 1400Release 3.11 (release-date: 2012-06-28) 1401======================================= 1402 1403New Features 1404------------ 1405 1406- Added support for configuring website buckets. 1407 1408Bug Fixes 1409--------- 1410 1411- Fixed bug that caused simultaneous resumable downloads of the same source 1412 object to use the same tracker file. 1413- Changed language code spec pointer from Wikipedia to loc.gov (for 1414 Content-Language header). 1415 1416 1417Release 3.10 (release-date: 2012-06-19) 1418======================================= 1419 1420New Features 1421------------ 1422 1423- Added support for setting and listing Content-Language header. 1424 1425Bug Fixes 1426--------- 1427 1428- Fixed bug that caused getacl/setacl commands to get a character encoding 1429 exception when ACL content contained content not representable in ISO-8859-1 1430 character set. 1431- Fixed gsutil update not to fail under Windows exclusive file locking. 1432- Fixed gsutil ls -L to continue past 403 errors. 1433- Updated gsutil tests and also help dev with instructions on how to run 1434 boto tests, based on recent test refactoring done to in boto library. 1435- Cleaned up parts of cp help text. 1436 1437 1438Release 3.9 (release-date: 2012-05-24) 1439====================================== 1440 1441Bug Fixes 1442--------- 1443 1444- Fixed bug that caused extra "file:/" to be included in pathnames with 1445 doing gsutil cp -R on Windows. 1446 1447 1448Release 3.8 (release-date: 2012-05-20) 1449====================================== 1450 1451Bug Fixes 1452--------- 1453 1454- Fixed problem with non-ASCII filename characters not setting encoding before 1455 attempting to hash for generating resumable transfer filename. 1456 1457 1458Release 3.7 (release-date: 2012-05-11) 1459====================================== 1460 1461Bug Fixes 1462--------- 1463 1464- Fixed handling of HTTPS tunneling through a proxy. 1465 1466 1467Release 3.6 (release-date: 2012-05-09) 1468====================================== 1469 1470Bug Fixes 1471--------- 1472 1473- Fixed bug that caused wildcards spanning directories not to work. 1474- Fixed bug that gsutil cp -z not to find available tmp space correctly 1475 under Windows. 1476 1477 1478Release 3.5 (release-date: 2012-04-30) 1479====================================== 1480 1481Performance Improvement 1482----------------------- 1483 1484- Change by Evan Worley to calculate MD5s incrementally during uploads and 1485 downloads. This reduces overall transfer time substantially for large 1486 objects. 1487 1488Bug Fixes 1489--------- 1490 1491- Fixed bug where uploading and moving multiple files to a bucket subdirectory 1492 didn't work as intended. 1493 (https://github.com/GoogleCloudPlatform/gsutil/issues/92). 1494- Fixed bug where gsutil cp -r sourcedir didn't copy to specified subdir 1495 if there is only one file in sourcedir. 1496- Fixed bug where tracker file included a timestamp that caused it not to 1497 be recognized across sessions. 1498- Fixed bug where gs://bucket/*/dir wildcard matches too many objects. 1499- Fixed documentation errors in help associated with ACLs and projects. 1500- Changed GCS ACL parsing to be case-insensitive. 1501- Changed ls to print error and exit with non-0 status when wildcard matches 1502 nothing, to be more consistent with UNIX shell behavior. 1503 1504 1505Release 3.4 (release-date: 2012-04-06) 1506====================================== 1507 1508Bug Fixes 1509--------- 1510 1511- Fixed problem where resumable uploads/downloads of objects with very long 1512 names would generate tracking files with names that exceeded local file 1513 system limits, making it impossible to complete resumable transfers for 1514 those objects. Solution was to build the tracking file name from a fixed 1515 prefix, SHA1 hash of the long filename, epoch timestamp and last 16 1516 chars of the long filename, which is guarantee to be a predictable and 1517 reasonable length. 1518- Fixed minor bug in output from 'gsutil help dev' which advised executing 1519 an inconsequential test script (test_util.py). 1520 1521 1522Release 3.3 (release-date: 2012-04-03) 1523====================================== 1524 1525Bug Fixes 1526--------- 1527 1528- Fixed problem where gsutil ver and debug flags crashed when used 1529 with newly generated boto config files. 1530- Fixed gsutil bug in windows path handling, and make checksumming work 1531 across platforms. 1532- Fixed enablelogging to translate -b URI param to plain bucket name in REST 1533 API request. 1534 1535 1536Release 3.2 (release-date: 2012-03-30) 1537====================================== 1538 1539Bug Fixes 1540--------- 1541 1542- Fixed problem where gsutil didn't convert between OS-specific directory 1543 separators when copying individually-named files (issue 87). 1544- Fixed problem where gsutil ls -R didn't work right if there was a key 1545 with a leading path (like /foo/bar/baz) 1546 1547 1548Release 3.1 (release-date: 2012-03-20) 1549====================================== 1550 1551Bug Fixes 1552--------- 1553 1554- Removed erroneous setting of Content-Encoding when a gzip file is uploaded 1555 (vs running gsutil cp -z, when Content-Encoding should be set). This 1556 error caused users to get gsutil.tar.gz file uncompressed by the user 1557 agent (like wget) while downloading, making the file appear to be of the 1558 wrong size/content. 1559- Fixed handling of gsutil help for Windows (previous code depended on 1560 termios and fcntl libs, which are Linux/MacOS-specific). 1561 1562 1563Release 3.0 (release-date: 2012-03-20) 1564====================================== 1565 1566Important Notes 1567--------------- 1568 1569- Backwards-incompatible wildcard change: 1570 The '*' wildcard now only matches objects within a bucket directory. If 1571 you have scripts that depend on being able to match spanning multiple 1572 directories you need to use '**' instead. For example, the command: 1573 1574 gsutil cp gs://bucket/*.txt 1575 1576 will now only match .txt files in the top-level directory. 1577 1578 gsutil cp gs://bucket/**.txt 1579 1580 will match across all directories. 1581- gsutil ls now lists one directory at a time. If you want to list all objects 1582 in a bucket, you can use: 1583 1584 gsutil ls gs://bucket/** 1585 1586 or: 1587 1588 gsutil ls -R gs://bucket 1589 1590New Features 1591------------ 1592 1593- Built-in help for all commands and many additional topics. Try 1594 "gsutil help" for a list of available commands and topics. 1595- A new hierarchical file tree abstraction layer, which makes the flat bucket 1596 name space look like a hierarchical file tree. This makes several things 1597 possible: 1598 - copying data to/from bucket sub-directories (see “gsutil help cp”). 1599 - distributing large uploads/downloads across many machines 1600 (see “gsutil help cp”) 1601 - renaming bucket sub-directories (see “gsutil help mv”). 1602 - listing individual bucket sub-directories and for listing directories 1603 recursively (see “gsutil help ls”). 1604 - setting ACLs for objects in a sub-directory (see “gsutil help setacl”). 1605 1606- Support for per-directory (*) and recursive (**) wildcards. Essentially, 1607 ** works the way * did in previous gsutil releases, and * now behaves 1608 consistently with how it works in command interpreters (like bash). The 1609 ability to specify directory-only wildcards also enables a number of use 1610 cases, such as distributing large uploads/downloads by wildcarded name. See 1611 "gsutil help wildcards" for details. 1612- Support for Cross-Origin Resource Sharing (CORS) configuration. See "gsutil 1613 help cors" for details. 1614- Support for multi-threading and recursive operation for setacl command 1615 (see “gsutil help setacl”). 1616- Ability to use the UNIX 'file' command to do content type recognition as 1617 an alternative to filename extensions. 1618- Introduction of new end-to-end test suite. 1619- The gsutil version command now computes a checksum of the code, to detect 1620 corruption and local modification when assisting with technical support. 1621- The gsutil update command is no longer beta/experimental, and now also 1622 supports updating from named URIs (for early/test releases). 1623- Changed gsutil ls -L to also print Content-Disposition header. 1624 1625Bug Fixes 1626--------- 1627 1628- The gsutil cp -t option previously didn't work as documented, and instead 1629 Content-Type was always detected based on filename extension. Content-Type 1630 detection is now the default, the -t option is deprecated (to be removed in 1631 the future), and specifying a -h Content-Type header now correctly overrides 1632 the filename extension based handling. For details see "gsutil help 1633 metadata". 1634- Fixed bug that caused multi-threaded mv command not to percolate failures 1635 during the cp phase to the rm phase, which could under some circumstances 1636 cause data that was not copied to be deleted. 1637- Fixed bug that caused gsutil to use GET for ls -L requests. It now uses HEAD 1638 for ls -L requests, which is more efficient and faster. 1639- Fixed bug that caused gsutil not to preserve metadata during 1640 copy-in-the-cloud. 1641- Fixed bug that prevented setacl command from allowing DisplayName's in ACLs. 1642- Fixed bug that caused gsutil/boto to suppress consecutive slashes in path 1643 names. 1644- Fixed spec-non-compliant URI construction for resumable uploads. 1645- Fixed bug that caused rm -f not to work. 1646- Fixed UnicodeEncodeError that happened when redirecting gsutil ls output 1647 to a file with non-ASCII object names. 1648 1649Other Changes 1650------------- 1651 1652- UserAgent sent in HTTP requests now includes gsutil version number and OS 1653 name. 1654- Starting with this release users are able to get individual named releases 1655 from version-named objects: gs://pub/gsutil_<version>.tar.gz 1656 and gs://pub/gsutil_<version>.zip. The version-less counterparts 1657 (gs://pub/gsutil.tar.gz and gs://pub/gsutil.zip) will contain the latest 1658 release. Also, the gs://pub bucket is now publicly readable (so, anyone 1659 can list its contents). 1660 1661 1662Release 2.0 (release-date: 2012-01-13) 1663====================================== 1664 1665New Features 1666------------ 1667 1668- Support for for two new installation modes: enterprise and RPM. 1669 Customers can now install gsutil one of three ways: 1670 - Individual user mode (previously the only available mode): unpacking from 1671 a gzipped tarball (gs://pub/gsutil.tar.gz) or zip file 1672 (gs://pub/gsutil.zip) and running the gsutil command in place in the 1673 unpacked gsutil directory. 1674 - Enterprise mode (new): unpacking as above, and then running the setup.py 1675 script in the unpacked gsutil directory. This allows a systems 1676 administrator to install gsutil in a central location, using the Python 1677 distutils facility. This mode is supported only on Linux and MacOS. 1678 - RPM mode (new). A RedHat RPM can be built from the gsutil.spec.in file 1679 in the unpacked gsutil directory, allowing it to be installed as part of 1680 a RedHat build. 1681 1682- Note: v2.0 is the first numbered gsutil release. Previous releases 1683 were given timestamps for versions. Numbered releases enable downstream 1684 package builds (like RPMs) to define dependencies more easily. 1685 This is also the first version where we began including release notes. 1686