<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/gitlab/gitlab-ce.git/spec/uploaders, branch docs/https-many-docs-links</title>
<subtitle>gitlab.com: gitlab-org/gitlab-ce.git
</subtitle>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/'/>
<entry>
<title>Remove legacy artifact related code</title>
<updated>2019-05-31T03:49:17+00:00</updated>
<author>
<name>Shinya Maeda</name>
<email>shinya@gitlab.com</email>
</author>
<published>2019-03-22T12:38:45+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=387a4f4b2cc9cffe2a21ef1060fe35c60d1ac769'/>
<id>387a4f4b2cc9cffe2a21ef1060fe35c60d1ac769</id>
<content type='text'>
We've already migrated all the legacy artifacts to the new realm,
which is ci_job_artifacts table.
It's time to remove the old code base that is no longer used.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
We've already migrated all the legacy artifacts to the new realm,
which is ci_job_artifacts table.
It's time to remove the old code base that is no longer used.
</pre>
</div>
</content>
</entry>
<entry>
<title>Fix incorrect prefix used in new uploads for personal snippets</title>
<updated>2019-05-17T06:36:55+00:00</updated>
<author>
<name>Stan Hu</name>
<email>stanhu@gmail.com</email>
</author>
<published>2019-05-15T16:26:18+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=a1f44c1b4969847fa80d6c53d6bd70813d273d6c'/>
<id>a1f44c1b4969847fa80d6c53d6bd70813d273d6c</id>
<content type='text'>
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24550 fixed the
case where the wrong path on disk was being searched, but it
inadvertently ommitted the `/uploads/-/system` prefix when rendering the
Markdown for personal snippet uploads when they were stored directly in
object storage.

A personal snippet path is stored using FileUploader#upload_path.
The format for the path:

Local storage: :random_hex/:filename.
Object storage: personal_snippet/:id/:random_hex/:filename.

upload_paths represent the possible paths for a given identifier,
which will vary depending on whether the file is stored in local or
object storage. upload_path should match an element in upload_paths.

base_dir represents the path seen by the user in Markdown, and it
should always be prefixed with uploads/-/system.

store_dirs represent the paths that are actually used on disk. For
object storage, this should omit the prefix /uploads/-/system.

For example, consider the requested path
/uploads/-/system/personal_snippet/172/ff4ad5c2/file.png.

For local storage:

base_dir: uploads/-/system/personal_snippet/172
upload_path: ff4ad5c2/file.png
upload_paths: ["ff4ad5c2/file.png", "personal_snippet/172/ff4ad5c2/file.png"].
store_dirs: {1=&gt;"uploads/-/system/personal_snippet/172/ff4ad5c2",
             2=&gt;"personal_snippet/172/ff4ad5c2"}

For object storage:

upload_path: personal_snippet/172/ff4ad5c2/file.png

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/61671
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24550 fixed the
case where the wrong path on disk was being searched, but it
inadvertently ommitted the `/uploads/-/system` prefix when rendering the
Markdown for personal snippet uploads when they were stored directly in
object storage.

A personal snippet path is stored using FileUploader#upload_path.
The format for the path:

Local storage: :random_hex/:filename.
Object storage: personal_snippet/:id/:random_hex/:filename.

upload_paths represent the possible paths for a given identifier,
which will vary depending on whether the file is stored in local or
object storage. upload_path should match an element in upload_paths.

base_dir represents the path seen by the user in Markdown, and it
should always be prefixed with uploads/-/system.

store_dirs represent the paths that are actually used on disk. For
object storage, this should omit the prefix /uploads/-/system.

For example, consider the requested path
/uploads/-/system/personal_snippet/172/ff4ad5c2/file.png.

For local storage:

base_dir: uploads/-/system/personal_snippet/172
upload_path: ff4ad5c2/file.png
upload_paths: ["ff4ad5c2/file.png", "personal_snippet/172/ff4ad5c2/file.png"].
store_dirs: {1=&gt;"uploads/-/system/personal_snippet/172/ff4ad5c2",
             2=&gt;"personal_snippet/172/ff4ad5c2"}

For object storage:

upload_path: personal_snippet/172/ff4ad5c2/file.png

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/61671
</pre>
</div>
</content>
</entry>
<entry>
<title>Optimise upload path calls</title>
<updated>2019-05-14T16:01:48+00:00</updated>
<author>
<name>Sean McGivern</name>
<email>sean@gitlab.com</email>
</author>
<published>2019-05-11T12:06:44+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=79dd92c8189600f24a29fe2da8d391cdb021f8fd'/>
<id>79dd92c8189600f24a29fe2da8d391cdb021f8fd</id>
<content type='text'>
String#underscore isn't particularly slow, but it's possible for us to
call it many times in a users autocomplete request, with mostly-static
values ('User', 'Group', etc.). We can memoise this and save a
surprising amount of time (around 10% of the total request time in some
cases).
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
String#underscore isn't particularly slow, but it's possible for us to
call it many times in a users autocomplete request, with mostly-static
values ('User', 'Group', etc.). We can memoise this and save a
surprising amount of time (around 10% of the total request time in some
cases).
</pre>
</div>
</content>
</entry>
<entry>
<title>Clean up CarrierWave's import/export files</title>
<updated>2019-05-07T08:27:50+00:00</updated>
<author>
<name>Stan Hu</name>
<email>stanhu@gmail.com</email>
</author>
<published>2019-04-18T12:10:21+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=ec341a2bbd81eac9639faf5c5e8e8b670f535096'/>
<id>ec341a2bbd81eac9639faf5c5e8e8b670f535096</id>
<content type='text'>
Unlike uploads that have been uploaded with Tempfile, the project
import/export archives are stored in a temporary cache directory and
remain there if:

1. Object storage is enabled
2. `move_to_store` is set to `true`.

CarrierWave will leave these files there until disk space runs out or a
clean step is run manually.

If `move_to_store` is set to `false`, CarrierWave will remove the files
after storing them. However, unlike a local file, with object storage,
the file is still copied, so setting `move_to_store` to `true`
doesn't buy us anything.

To ensure files are cleaned up, we can just inherit from the
GitlabUploader implementation of `move_to_store`, which returns `true`
if it's a local file, `false` otherwise.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/60656
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Unlike uploads that have been uploaded with Tempfile, the project
import/export archives are stored in a temporary cache directory and
remain there if:

1. Object storage is enabled
2. `move_to_store` is set to `true`.

CarrierWave will leave these files there until disk space runs out or a
clean step is run manually.

If `move_to_store` is set to `false`, CarrierWave will remove the files
after storing them. However, unlike a local file, with object storage,
the file is still copied, so setting `move_to_store` to `true`
doesn't buy us anything.

To ensure files are cleaned up, we can just inherit from the
GitlabUploader implementation of `move_to_store`, which returns `true`
if it's a local file, `false` otherwise.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/60656
</pre>
</div>
</content>
</entry>
<entry>
<title>Disable method replacement in avatar loading</title>
<updated>2019-04-30T04:29:25+00:00</updated>
<author>
<name>Stan Hu</name>
<email>stanhu@gmail.com</email>
</author>
<published>2019-04-30T00:07:42+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=25818bd7ae765422c934d0a32efb4ba353d11183'/>
<id>25818bd7ae765422c934d0a32efb4ba353d11183</id>
<content type='text'>
We've seen a significant performance penalty when using
`BatchLoader#__replace_with!`. This defines methods on the batch loader
that proxy to the 'real' object using send. The alternative is
`method_missing`, which is slower.  However, we've noticed that
`method_missing` can be faster if:

1. The objects being loaded have a large interface.
2. We don't call too many methods on the loaded object.

Avatar uploads meet both criteria above, so let's use the newly-released
feature in https://github.com/exAspArk/batch-loader/pull/45.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/60903
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
We've seen a significant performance penalty when using
`BatchLoader#__replace_with!`. This defines methods on the batch loader
that proxy to the 'real' object using send. The alternative is
`method_missing`, which is slower.  However, we've noticed that
`method_missing` can be faster if:

1. The objects being loaded have a large interface.
2. We don't call too many methods on the loaded object.

Avatar uploads meet both criteria above, so let's use the newly-released
feature in https://github.com/exAspArk/batch-loader/pull/45.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/60903
</pre>
</div>
</content>
</entry>
<entry>
<title>Autocorrect with RSpec/ExampleWording cop</title>
<updated>2019-04-05T08:43:27+00:00</updated>
<author>
<name>Thong Kuah</name>
<email>tkuah@gitlab.com</email>
</author>
<published>2019-04-05T08:43:27+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=4ec16912b8f4eba9cc70dd21afb65c441f857da0'/>
<id>4ec16912b8f4eba9cc70dd21afb65c441f857da0</id>
<content type='text'>
- rewords examples starting with 'should'
- rewords examples starting with 'it'

Note: I had to manually fixup "onlies" to "only"
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
- rewords examples starting with 'should'
- rewords examples starting with 'it'

Note: I had to manually fixup "onlies" to "only"
</pre>
</div>
</content>
</entry>
<entry>
<title>Speed up avatar URLs with object storage</title>
<updated>2019-04-04T10:32:42+00:00</updated>
<author>
<name>Sean McGivern</name>
<email>sean@gitlab.com</email>
</author>
<published>2019-04-02T11:52:28+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=ec85debaf51067cc78d54188ec1eef94342d5a8b'/>
<id>ec85debaf51067cc78d54188ec1eef94342d5a8b</id>
<content type='text'>
With object storage enabled, calling `#filename` on an upload does this:

1. Call the `#filename` method on the CarrierWave object.
2. Generate the URL for that object.
3. If the uploader isn't public, do so by generating an authenticated
   URL, including signing that request.

That's all correct behaviour, but for the case where we use `#filename`,
it's typically to generate a GitLab URL. That URL doesn't need to be
signed because we do our own auth.

Signing the URLs can be very expensive, especially in batch (say, we
need to get the avatar URLs for 150 users in one request). It's all
unnecessary work. If we used the `RecordsUploads` concern, we have
already recorded a `path` in the database. That `path` is actually
generated from CarrierWave's `#filename` at upload time, so we don't
need to recompute it - we can just use it and strip off the prefix if
it's available.

On a sample users autocomplete URL, at least 10% of the time before this
change went to signing URLs. After this change, we spend no time in URL
signing, and still get the correct results.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
With object storage enabled, calling `#filename` on an upload does this:

1. Call the `#filename` method on the CarrierWave object.
2. Generate the URL for that object.
3. If the uploader isn't public, do so by generating an authenticated
   URL, including signing that request.

That's all correct behaviour, but for the case where we use `#filename`,
it's typically to generate a GitLab URL. That URL doesn't need to be
signed because we do our own auth.

Signing the URLs can be very expensive, especially in batch (say, we
need to get the avatar URLs for 150 users in one request). It's all
unnecessary work. If we used the `RecordsUploads` concern, we have
already recorded a `path` in the database. That `path` is actually
generated from CarrierWave's `#filename` at upload time, so we don't
need to recompute it - we can just use it and strip off the prefix if
it's available.

On a sample users autocomplete URL, at least 10% of the time before this
change went to signing URLs. After this change, we spend no time in URL
signing, and still get the correct results.
</pre>
</div>
</content>
</entry>
<entry>
<title>Don't try to set any ACL on uploaded objects</title>
<updated>2019-03-30T13:55:33+00:00</updated>
<author>
<name>Bastian Blank</name>
<email>waldi@debian.org</email>
</author>
<published>2019-03-29T22:54:44+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=b82ccf2cbc074e36c78dd1e331c6745cb9b71349'/>
<id>b82ccf2cbc074e36c78dd1e331c6745cb9b71349</id>
<content type='text'>
Set ACL of uploaded objects to not-public (fog-aws) or no ACL at all
(fog-google).  Value is ignored by other supported backends (fog-aliyun,
fog-openstack, fog-rackspace)

This allows uploads to GCS with ACL support disabled.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Set ACL of uploaded objects to not-public (fog-aws) or no ACL at all
(fog-google).  Value is ignored by other supported backends (fog-aliyun,
fog-openstack, fog-rackspace)

This allows uploads to GCS with ACL support disabled.
</pre>
</div>
</content>
</entry>
<entry>
<title>Check snippet attached file to be moved is within designated directory</title>
<updated>2019-02-21T08:44:44+00:00</updated>
<author>
<name>Mark Chao</name>
<email>mchao@gitlab.com</email>
</author>
<published>2019-02-13T08:24:26+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=d72b1cd0b5b01d6fec6b93d9dfe84f8302083072'/>
<id>d72b1cd0b5b01d6fec6b93d9dfe84f8302083072</id>
<content type='text'>
Previously one could move any temp/ sub folder around.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Previously one could move any temp/ sub folder around.
</pre>
</div>
</content>
</entry>
<entry>
<title>Align spec with actual usage</title>
<updated>2019-02-13T08:50:31+00:00</updated>
<author>
<name>Mark Chao</name>
<email>mchao@gitlab.com</email>
</author>
<published>2019-02-13T08:23:58+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=a9291f15ea10e3cfc94282ffb4e0969e9d4175eb'/>
<id>a9291f15ea10e3cfc94282ffb4e0969e9d4175eb</id>
<content type='text'>
Currently we pass temp file path to FileMover
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Currently we pass temp file path to FileMover
</pre>
</div>
</content>
</entry>
</feed>
