Unable to set file permissions in S3 using boto and django

Unable to set file permissions in S3 using boto and django

I have been trying to find a solution to this for about 36 hours now so
hopefully I'm not duplicating a question or asking something obvious. I am
building a web app that has to manipulate files that I store in S3 and put
back the new versions in S3 with a 'public-read' acl. Then a different
page allows you to view the updated file. The app exists on an amazon EC2
server and connects to an amazon S3 bucket.
I am using django, celery, and boto to do this. I have a celery task set
up that gets some info from one of my views and does the processing and
then posts the new file to S3. I am able to get the original file from S3,
manipulate it successfully, and repost it to S3. The only thing that
doesn't seem to work is changing the permissions on that file. So
everything works except when you go to the viewing page, I get a 403
(Forbidden) error when trying to access that file.
If I go into S3 myself and change the permission on that file for everyone
to read, it all works. Before I go on, the code that I use in my task that
almost works is:
name = 'filename.blah'
conn = boto.connect_s3()
b = conn.get_bucket(settings.AWS_STORAGE_BUCKET_NAME)
grab_from_S3(name,b) # grab file from S3
out_name = conv(name)
send_to_S3(out_name,b)
where the functions in there are:
def grab_from_S3(file,bucket):
k = Key(bucket)
k.key = file
k.get_contents_to_filename(file)
def send_to_S3(file,bucket):
k = Key(bucket)
k.key = file
k.set_contents_from_filename(file)
k.set_acl('public-read')
and conv(name) just does some conversion stuff. So this works almost all
the way except the file's permissions are not 'public-read'. All the AWS
credentials and bucket name I assume are being imported from the
environment properly because it is able to push and pull files to and from
S3.
The big confusing part is that when I open up a python environment from
either the venv on my EC2 server or just the python that was installed on
it to begin with and I run all the commands I show above, it DOES work. I
can change the permission with no problems. And when the task runs it does
not throw any errors in the celery logs so I don't think the task is
actually running into errors. It's just simply not changing what it's
supposed to change.
Things that I have tried:
I tried to use other versions of the permissions function such as
k.set_contents_from_filename(file,policy='public-read') or k.make_public()
or b.set_acl('public-read',out_name) but none of those worked either.
I changed the permissions on the bucket to say that everyone was allowed
to change the permissions and it didn't work still.
I tried to change the bucket policy to this below (sorry I don't know how
to format it right) and it made no effect:
{ "Version": "2008-10-17", "Id": "whatever", "Statement": [ { "Sid":
"whatever", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": [
"s3:PutObjectAcl", "s3:PutObject" ], "Resource": [
"arn:aws:s3:::bucket_name", "arn:aws:s3:::bucket_name/*" ] } ] }
In the end, I'm really confused because I can seem to do all of this just
fine from a python environment on the same EC2 instance but not the code
running on that instance. I've searched and searched and haven't been able
to find any suggestions that worked. Another possibly useful piece of info
(but it might be irrelevant depending on the problem) is that if I try to
connect to S3 in my view by doing similar commands above it returns an
error:
"No handler was ready to authenticate. 1 handlers were checked.
['HmacAuthV1Handler'] Check your credentials"
even though it works when those commands are run in my task (I would
assume it was the wrong access key or secret access key or something, but
it works with everything else). I think I'm doing the correct imports in
the python code of the parts of the boto library I need.
I just recently set this instance up so it has probably almost the newest
version of boto, celery, django, etc on it. I probably forgot something.
Please let me know if you need more info to answer the question. I'm
really not sure what is going on.
Thanks a ton in advance.